Two goals from Jordan Rhodes - his 31st and 32nd of the season - and one from Sean Morrison seemed to have left the hosts dead and buried inside 29 minutes.
But Steve Schumacher and Lenell John-Lewis each pulled one back before Ashley Eastham volleyed home Peter Sweeney's corner in injury time.
Huddersfield claimed three goals in eight first-half minutes, the first coming when Morrison slammed home from Danny Ward's corner before Rhodes grabbed a quick-fire double with a header and a thumping shot.
Not to be outdone, Schumacher struck a superb 25-yard free kick into the bottom corner to get his side off the mark.
David Amoo was nimbly denied by Huddersfield goalkeeper Alex Smithies soon after the break, while Ward struck the crossbar when clean through on goal, but Bury halved the deficit when Amoo crossed for John-Lewis to clip a shot past Smithies and Eastham lashed home a stoppage-time corner to become a last-gasp hero.
