In the 82nd minute of Tuesday night's clash at Gigg Lane, David Worrall drilled the ball into the box and Craig Jones guided home his first senior goal from eight yards.
Then with just a minute left, substitute Johnson Clarke-Harris sprinted clear of the Stevenage defence and buried a low shot into the corner.
Bury created the better chances - and missed a penalty - in a goalless first half.
In the 20th minute, Gavin Mahon tripped Steven Schumacher just inside the box, but after the Bury skipper dusted himself down, his subsequent penalty kick was superbly saved down low to his left by goalkeeper Steven Arnold.
At the other end, Dani Lopez saw a strike well saved by Trevor Carson, before Bury missed two chances in rapid succession to break the deadlock just before the break.
Andy Bishop's initial shot was saved by Arnold, before Jones volleyed the rebound over from six yards.
Lucas Akins went closest for the visitors in the second half before Bury eventually struck.