Gavin Massey put Colchester ahead after just two minutes, but Jed Wallace's volley levelled the scores three minutes later.
Sam Sodje's clumsy tackle on Freddie Sears handed Colchester a penalty, which Sears coolly dispatched after 15 minutes before Massey's deflected strike made it 3-1 to Colchester five minutes later.
James Keene pulled one back for Pompey after 73 minutes, but their second-half pressure wasn't enough to prevent a seventh straight defeat.
Dunne hailed the U's resilience as they picked up a second consecutive league victory.
He said: "We've spoken about the fact we haven't scored enough goals, but we've been working exceptionally hard in training defensively and offensively and we're glad that has paid off.
"I thought we were resolute. A few weeks ago, we probably would have crumbled.
"As much as it's really pleasing to have scored three goals away from home, and we should have scored more, we can take as much from how we defended in the second half and how resolute we were.
"It's back-to-back wins, but we have to keep on playing. We have to keep on working hard. We can't rest now or else it will all fall around us again.
"Great credit to the players. We came off the back of nine defeats and now it's back-to-back wins. How quickly football changes."
