Wes Fogden scored in the third minute of injury time to clinch an unlikely point after Hallam Hope’s hat-trick had put Bury in command.
The hosts had raced out of the blocks and hit the front when on-loan Everton striker Hope put a looping header into the top corner on nine minutes.
Hope grabbed his second in the 34th minute, picking up Tom Soares’ pass before planting a superb right-footed finish into the bottom corner.
Bury’s third followed four minutes into the second half when Andy Procter netted a diving header from Danny Mayor’s inviting cross.
But Danny Hollands fired home after Fogden’s shot had been blocked in the 53rd minute to make it 3-1.
Taylor cut the deficit further in the 82nd minute, putting away Patrick Agyemang’s cut-back but Bury seemed to wrap up victory when Hope sealed his hat-trick with a superb strike five minutes later.
However, substitute Sonny Bradley poked home from a corner in the 88th minute and Fogden slammed Hollands’ long throw into the net from close range in stoppage time.