The striker's first league goal for Swindon came within the opening minute, giving Paolo Di Canio's side a lead they never relinquished.
It was a goal worthy of winning any game, Collins beating Bury goalkeeper Trevor Carson with a stunning 20-yard lob.
Bury rallied and were denied an equaliser by a superb challenge from Jay McEveley just as David Healy got ready to pull the trigger.
Despite applying a decent amount of pressure, Bury struggled to find a way through, Peter Sweeney lashing a long-range effort over the bar.
But Swindon, for the most part, controlled the second half and Collins wasted a chance for his second by blasting over from inside the box, while Carson produced a tremendous save to push away a stinging volley from Adam Rooney.