Pompey led twice through Patrick Agyemang and Johnny Ertl, but Jermaine McGlashan levelled twice, with his second arriving in the 92nd minute.
"Our first-half performance was decent enough, but in the second half it was backs to the wall and we didn't keep the ball well enough with their pressure eventually telling," Whittingham said.
"In the first half we created a lot of good chances, but we don't seem to keep playing the football we are capable of.
"We've had a discussion about it, with calm heads and we have to make sure we get it right. Three times we've been winning away and not been able to hold out.
"If you look at the games we've conceded, we've defended well on the whole so it's frustrating to let one in at the end, but we'll get it right.
"You are more worried if you are not playing well, but we are and it's just the goals going in that are costing us.
"We have to finish games off and go two goals ahead, which we should have been early in the second half."
Portsmouth had taken the lead through Agyemang in the fifth minute. He capitalised on Steve Elliott's stumble to poke the ball past goalkeeper Scott Brown from close range.
Agyemang should have been celebrating again in the 13th minute when he blazed over with the goal gaping after a fine run from Yassin Moutaouakil.
Portsmouth held the upper hand for most of the half, but the introduction of Russ Penn from the bench injected life into Cheltenham.
He replaced the injured Ryan Inniss and soon after a reshuffle, Terry Gornell set up McGlashan for the leveller in the 36th minute.
The quick winger rounded Sullivan and applied a neat finish inside the bottom right corner.
Cheltenham pressed for a second, enjoying their best period of the half, but Portsmouth regained the lead in the 43rd minute.
Jed Wallace reached the byline and pulled back for Ertl to tap into an empty net and it looked to be enough until McGlashan struck at the death.