Pompey striker Izale McLeod had a penalty stopped by Orient goalkeeper Ryan Allsop, who felled Brian Howard, after 26 minutes.
The visitors took charge just six minutes later when Dean Cox brilliantly slotted in Kevin Lisbie, who coolly swept his shot beyond Mikkel Andersen.
Orient doubled their lead after 52 minutes when Moses Odubajo slipped into space in the Pompey box and lashed the ball home. McLeod atoned for his spot-kick failure four minutes later when Pompey's Paul Benson was felled by Nathan Clarke in the box.
But Orient regained their two-goal cushion when referee Graham Scott awarded them the game's third penalty after 61 minutes when Lisbie was brought down by the returning Ricardo Rocha. Lisbie stroked confidently past Andersen.
Pompey pulled another one back after 65 minutes when Scott Allan's shot was deflected beyond Allsop by Martin Rowlands. The home side launched a relentless assault on Orient in the closing stages, but could not snatch what would have been a well-deserved point.
