Armand Gnauduillet's header in the 87th minute and Tendayi Darikwa's strike two minutes into stoppage time snatched the Spireites their fourth League Two win in their opening five games.
But Pompey's hopes of victory suffered a huge blow seven minutes before the break when Connolly was sent off for an off-the-ball incident with Chesterfield's Sam Hird.
The clash came just moments after Connolly had strong claims for a penalty turned down by referee Stephen Bratt after he was tripped in the box as he raced on to Simon Ferry's through ball.
Whittingham admits Connolly was frustrated after being denied a spot kick, but was surprised by his confrontation with Hird, which was punished after assistant referee Michael Webb witnessed it.
He said: "David has apologised for it, but I didn't see the incident. Of course, you are a little bit surprised by it. He's a seasoned pro.
"I think he was wound up by one or two decisions he didn't get earlier. That's no excuse, but the referee is probably the only one who didn't think it was a penalty, Even the Chesterfield staff said it was a stonewall penalty."
Whittingham's men still looked the more likely of the teams to score until the closing stages and he hailed Pompey's fighting spirit.
He said: "I think that's what we need to focus on - not the fact David was sent off. The fans certainly appreciated the spirit the team showed.
"We all feel disappointed we didn't come out of the game with any points, but we have showed people there is a real desire and spirit to do well in front of our great fans.
"The positives outweigh the negatives. We fought for everything. We created chances even though we were down to 10 men."