The hosts went in front after eight minutes as Craig Reid teed up Scott Laird and he crashed home low into the corner, but that lead lasted just seven minutes as Steven Schumacher fired home a crisp strike past Chris Day from 20 yards.
Stevenage quickly regained the lead though, as this time Bury keeper Cameron Belford tripped Byron Harrison in the box and Reid slotted home the resultant penalty kick.
The first half scoring wasn't over though, as Bury levelled again when Nicky Ajose comfortably nodded home from a Mike Jones cross.
With 20 minutes remaining it looked as though Bury might go on and snatch victory as Schumacher fired home another unstoppable shot from distance, but Stevenage weren't finished.
They made it 3-3 four minutes later when sub Ben May pounced from close range following Robert Sinclair's ball across goal.