The Glovers took the lead after 15 minutes when Pompey failed to clear a Jamie McAllister free-kick and Dominic Blizzard turned the loose ball past goalkeeper Simon Eastwood.
They doubled their advantage in almost identical fashion after 38 minutes when Pompey's rearguard did not deal with another McAllister set piece - this time Byron Webster hooked the ball home.
Pompey's players went down the tunnel for half-time with vociferous boos ringing in their ears.
Caretaker boss Guy Whittingham reacted by hauling off ineffective duo Izale McLeod and Brian Howard and bringing on Scott Allan and local favourite Ashley Harris.
The hosts pulled a goal back after 54 minutes when Liam Walker's cross from the right was hooked back across goal by Adam Webster at the far post, putting it on a plate for Paul Benson to score.
Benson nearly levelled with seven minutes left when he turned Walker's cross on to a post, but a spirited fightback was not quite enough to earn Pompey a point.
