Jose Baxter's early goal was enough to sink rock-bottom Pompey, who had Sam Sodje sent off for violent conduct just after half-time.
Following the 1-0 loss, Whittingham said: "We knew Oldham would cause us a problem if we didn't pressure them, but that's what happened.
"It was only after they scored that the players started doing what they were supposed to do, so the slow start has cost us.
"Our keeper has made a mistake and would usually stop that shot, but Simon (Eastwood) has done well for us this season and we move on.
"We were always in the game, including with 10 men, but we didn't get enough balls or people into the right areas."
Whittingham revealed that Sodje had apologised to his team-mates for the off-the-ball clash that earned him a red card.
"He has come in and said sorry, but that's no good to us," Whittingham said.
"There was a bit of an incident and everyone got excited, but I genuinely didn't see what happened. I'll have to look at the DVD and see it for myself."
