In the fourth minute George Porter was sent off for the U's, who's squad was hit by a virus this week, and Craig Fagan fired Bury into a 26th-minute lead.
Yet the visitors stunned the Shakers with two goals in two minutes from Josh Thompson and Clinton Morriston before surviving a second-half onslaught for their first win in four.
"It was a remarkable win in every sense of the word," said Dunne. "We had five players playing that were 50 per cent fit and had to leave five or six at home.
"They gave it their all and we rode our luck. But for all their chances, we'll take the win because I thought our courage and determination deserved something from the game.
"On another day, we would probably have lost. But we felt we could score against them and once we got the two goals, we would have taken a point. Our keeper and our back four were superb."
On the sending-off, Dunne added: "Apparently it's a red card and it's clean cut.
"It was a bit of a silly thing to do but the players were galvanised and it brought us the win.
"They started the game really well but I thought our response was superb."
