Caley got off to a dream start in the seventh minute when striker Adam Rooney had his flick deflected past Fraser Forster by Hoops' defender Charlie Mulgrew, before Kris Commons levelled less than two minutes later.
It looked like the visitors would reassert themselves to win their game in hand over the Light Blues and leapfrog them at the top of the table.
However, in a remarkable second half, goals from Grant Munro and Shane Sutherland - before Commons scored an injury-time penalty - gave the Highlanders an unlikely victory to keep Rangers one point clear of Celtic with three games remaining.
It was a night when most people expected Neil Lennon's side to take care of business and they certainly started off confidently.
Celtic racked up four corners in the first four minutes, all taken by Shaun Maloney on his first start since November, and it looked ominous for Terry Butcher's men.
The home side survived the early pressure, though, and with almost their first foray up the park in the seventh minute they took the lead.
Sutherland got behind the Celtic defence and his cross from the right was flicked on by Rooney, with the ball taking a touch off Mulgrew before flying past Forster.
However, the home side's lead was shortlived as, while the Caley supporters were still celebrating, Commons sent a drive from the edge of the box sailing past the helpless Ryan Esson.
It was frantic stuff and moments later, when Rooney broke through the Celtic rearguard with some ease, Forster was forced to make a blocking save.
At the other end, a low 30-yard drive from Celtic midfielder Beram Kayal was comfortably saved by Esson.
Celtic unsurprisingly had the edge in possession and territory and started to press Caley back as the first half unfolded.
The Highlanders defended resolutely and in numbers and frustrated the likes of Commons, Maloney, Gary Hooper and Anthony Stokes.
However, in the 32nd minute the Inverness defence were glad to see Common's left-footed drive from the edge of the box slip inches past Esson's left-hand post.
Any time Caley did attack they looked dangerous, especially when Rooney got involved, but Forster remained untroubled.
In the final minute of the half Esson made a fine fingertip save from Maloney's 25-yard free-kick which was heading just under the bar.
Inverness came out in determined mood at the start of the second half and in the 52nd minute their lead was restored by defender Munro, who had scored at Celtic Park earlier in the season.
Aaron Doran's corner from the right was only cleared to the edge of the box and Munro, one of 10 players who will be leaving Caley at the end of the season, took a touch before sending a wonderful volley past Forster.
Celtic appeared stunned and in the 58th minute Daryl Murphy replaced Stokes as Lennon tried another route back in to the game.
But in the 62nd minute there was bedlam inside the Tulloch Caledonian stadium as Caley increased their lead.
Celtic failed to clear a Doran free-kick properly and when Rooney returned the ball into the area, an unmarked Sutherland sent a low volley with his left foot past Forster and in off the far post.
Moments later, Esson made a good save from Commons' drive which came through a ruck of players, before another good save from Kayal's deflected shot.
In between, Paddy McCourt replaced Hooper as Lennon looked for some inspiration from the Irishman.
Ki Sung-yueng then replaced Kayal, who came off in some discomfort with what looked like a serious arm injury.
The visitors looked disjointed as Caley piled forward looking for a fourth and in the 75th minute Forster was again the hero when he thwarted Rooney, with Doran unable to control his shot from the rebound.
In the 79th minute Forster made a save from Nick Ross after the visiting defence was cut wide open again.
As Celtic pushed forward desperately, Daniel Majstorovic directed a Commons' cross past the far post.
In the second minute of added time Ross Tokely clumsily brought down Maloney in the box and Commons drove in the spot kick to give the visiting fans behind the goal some hope.
Celtic had another penalty claim turned down seconds later by referee Stevie O'Reilly when McCourt went to ground inside a packed the box as he went past Munro, and Caley hung on for the final few seconds to become the toast of Ibrox.