Hayden Mullins scored his second goal in three games to put Pompey ahead in the 17th minute but the Hornets hit back with two second-half goals from Martin Taylor and substitute Stephen McGinn.
Yet the spirited hosts drew level through on-loan Stoke forward Dave Kitson and Brown struck a long-range winner with 10 minutes remaining.
Both sides began the match in lively fashion and in the third minute Watford went close to opening the scoring when defender Adrian Mariappa found the space to hit a shot from 12 yards which flew over the crossbar.
Yet Pompey steadied themselves and fashioned a decent opening in the ninth minute when midfielder Liam Lawrence drilled a right-foot shot wide from 12 yards.
Lawrence went close again five minutes later before Portsmouth's pressure told in the 17th minute when Mullins struck from distance.
Having collected a pass from Lawrence, Mullins unleashed a superb right-foot shot which flew into the top corner from 30 yards out.
The Hornets were dealt another blow moments later when midfielder Don Cowie succumbed to injury and was replaced by McGinn.
Watford midfielder John Eustace tried his luck from distance in the 26th minute and Jordon Mutch also went close with a 30-yard strike.
However, the hosts were not without chances themselves and midfielder Nadir Ciftci hit a swerving 35-yard shot which sailed narrowly past Scott Loach's goal.
Shortly before half-time, David Nugent rifled a shot from 18 yards just wide as Pompey failed to find a second goal.
Watford grew as an attacking force after the restart as Danny Graham drew a smart stop from Jamie Ashdown in the 50th minute when his 12-yard strike was repelled from the Pompey goalkeeper.
In the 54th minute, Taylor showed his eye for goal with a 12-yard shot which drew another decent save from Ashdown.
In the 66th minute, Watford drew level when Taylor headed home unmarked from a corner by Mutch.
Three minutes later, the visitors were in front as McGinn connected with Michael Bryan's cross and fired home right-footed from six yards out.
Remarkably, Portsmouth were level two minutes later as Kitson collected possession 20 yards from goal and fired a low right-foot shot into the bottom corner.
And with 10 minutes remaining, Fratton Park went wild as Portsmouth scored a winning goal through Brown, who fired a right-foot into the bottom corner from 18 yards.