Kevin Lisbie put the visitors ahead after 32 minutes, six minutes after Pompey striker Izale McLeod had a penalty saved by Orient stopper Ryan Allsop.
The London club doubled their advantage through Moses Odubajo after 52 minutes, but McLeod was handed a second chance to score from the spot four minutes later and blasted the penalty home.
Lisbie won a spot-kick for the away side after 61 minutes and stepped up to fire it low past Mikkel Andersen. Scott Allan pulled another back for the hosts after 65 minutes as Pompey laid siege to the Orient goal.
After Tuesday night's League One clash at Fratton Park, Whittingham said: "It doesn't take a genius to see that the players are low on confidence. But that doesn't excuse anyone from working hard and we didn't do that in the first half.
"There were some choice words at half time. Andy Awford (Whittingham's assistant coach) got angrier than me. It was just a case of telling them that they are professional footballers, they represent this great club and they needed to make sure they went out in the second half and worked like professional footballers."