The visitors struck twice in four first-half minutes with Andy Barcham and then David Connolly both netting.
Substitute Lindon Meikle denied Portsmouth all three points with 19 minutes remaining and Whittingham was frustrated by the second-half display.
"The lads are a bit dejected in the dressing room that we did not win that game," he said.
"We are a big scalp for teams at this level and Mansfield put us under a lot of pressure. They do not make it an easy place to come.
"It was important that we hit back after falling behind and it was a good response from the boys.
"Andy Barcham looked very dangerous for us, he has been doing that all season. He is an exciting player who can score goals.
"The second half was frustrating for us. We felt we defended set pieces very well but one error cost us. We feel we should have held on to our win."