Akpo Sodje, playing against his elder brother Sam in the Pompey defence, opened the scoring in the 40th minute with a near-post header from a left-wing cross.
After Liam Walker had levelled for Whittingham's side on the stroke of half-time with a stunning 30-yard free-kick, it was Sodje again who popped up with an 85th-minute winner.
Whittingham said: "I thought we started brightly but then fell away in the first half. We have some young players in the side and they have to step up to the mark.
"There were some harsh words said in the dressing room at half-time but they reacted fantastically well in the second half.
"On our second-half performance alone we certainly didn't deserve to lose the game.
"I don't often have a go at referees but I thought this one had a poor game and we could have had a penalty in the second half. But the referee was not to blame for the late goal we conceded. That was down to poor defending.
"Both the goals we conceded were poor ones from our point of view. That was disappointing but we just have to battle on and hope we can find that bit of luck that can change things for us."
