The former England boss made 10 changes to his team at Gigg Lane on Tuesday night and almost paid the price, Mike Jones and Ryan Lowe putting the hosts 2-1 up.
But Paul Gallagher equalised to settle nerves before Lloyd Dyer and Neil Danns completed the comeback.
An unfamiliar-looking Foxes had taken some time to get going, with captain Matt Mills the only survivor from their 2-2 draw with Nottingham Forest.
Still, Jeff Schlupp put the visitors ahead with their first real chance.
Receiving the ball on the edge of the area, the 18-year-old turned sharply to fire low into the corner of the net.
Peter Sweeney almost equalised with a curling, left-foot free kick before Leicester allowed two good opportunities to slip away.
Yuki Abe hit the outside of the post from 15 yards and Schlupp was denied a second by the swift reactions of Cameron Belford.
The misses proved costly as Bury hit back to level five minutes before the interval.
Damien Mozika headed across goal from David Worrall’s cross and Lenell John-Lewis had an effort blocked before Jones lashed home from 18 yards.
It only got worse for Leicester after the break, Lowe whipping a delightful free-kick past Chris Weale from 20 yards to raise the spectre of an upset.
But Bury conceded their advantage when Efe Sodje upended Lee Peltier on the edge of the box and Gallagher blasted his free-kick under the wall and past Belford.
Sodje, so often a rock at the back, was at fault again for the decisive goal, conceding possession 20 yards out. Matt Oakley fed Dyer, who curled a truly outstanding finish into the far corner.
Dyer turned provider as Danns made sure of the win in injury time.