Rangers, who dominated for long periods, were leading with six minutes to go through a penalty from Adel Taarabt.
But as the game moved into stoppage time, Rangers failed to deal with Jamal Campbell-Ryce's corner and on-loan Tottenham defender Caulker crashed a header into the net.
Despite bossing the first half, Rangers somehow found themselves behind just after the interval as Clint Hill's mistake allowed Brett Pitman to convert a rare City chance.
However Neil Warnock's side, who had lost three of their last five matches having previously gone 19 games unbeaten to set the early pace, hit straight back when Taarabt set up Alejandro Faurlin.
Taarabt was at the heart of a one-sided opening period and fizzed an early 30-yard drive just over the top.
The hosts should have taken the lead in the ninth minute when Tommy Smith, who completed his permanent switch from Portsmouth on Monday morning, crossed for Rob Hulse but the big striker put his free header over.
Rangers had the ball in the net in the 18th minute but Jamie Mackie's celebrations were cut short by an offside trap.
David James had no chance with that effort but the 40-year-old former England keeper showed his class moments later with a stunning point-blank save to keep out another Hulse header.
Home keeper Paddy Kenny, a mere spectator for the first half an hour, was finally called into action when he kept out a low drive from Campbell-Ryce.
Normal service was soon resumed with Mackie and Taarabt firing wide, until City woke up suddenly produced the move of the first half, Campbell-Ryce and Jon Stead combining to put Pitman through only for Kenny to smother his shot.
There was still time for Hulse to blaze over yet again following a goalmouth scramble before the interval.
And four minutes after the break Rangers found themselves behind courtesy of a mistake from Hill, whose weak backward header was nowhere near Kenny and allowed Pitman to stroke the ball home for his eighth goal in eight games.
Stung into action, the hosts hit back within three minutes with that man Taarabt the architect.
The Moroccan teased Cole Skuse on his way to the byline and then slipped in a perfect pull-back for Faurlin to slot home from 10 yards.
City looked like holding out for a point until the 84th minute when referee Fred Graham deemed Elliott had held back Hulse and Taarabt did the rest from the penalty spot.
But in a dramatic late twist Caulker rose highest at the far post to head in Campbell-Ryce's cross to salvage a point.