In a pulsating match played in the right spirit, Steve Davis gave Rangers the lead in the 23rd minute before Joe Ledley levelled for the Hoops on the half-hour mark.
Three penalty claims - two for Rangers - were then ignored by referee Craig Thomson before Jelavic struck seven minutes in to extra-time to earn the Ibrox side the first silverware of the season.
In the dying seconds, Celtic left-back Emilio Izaguirre was shown a straight red for denying Vladimir Weiss a clear goalscoring opportunity but the game will be remembered for the right reasons.
After the Scottish Cup replay at Celtic Park earlier in the month which had ended with three Rangers players sent off and Hoops manager Neil Lennon and Ibrox assistant Ally McCoist having to be separated, police had visited both camps in midweek to warn them about their behaviour.
Both sets of players and management appeared to take heed.
With Lennon serving a touchline ban after being sent to the stand at Tynecastle in November, assistant Johan Mjallby led the teams out alongside Walter Smith, making his last appearance as Rangers boss at a cup final before he steps down at the end of the season.
After the formalities came a frantic and disjointed start to the game.
The first moment of controversy came in the 15th minute when Mark Wilson went crashing to the ground inside the penalty box after a challenge by Rangers Sasa Papac but referee Thomson was unimpressed.
Izaguirre was finding some joy down the left flank as Kyle Lafferty struggled to keep track
However, in the 21st minute it was Kris Commons' cross from wide on the left which Parkhead skipper Scott Brown dived to meet eight yards out but the ball flew past Neil Alexander's right-hand post.
Two minutes later that miss was compounded when Celtic found themselves behind.
Davis still had it all to do when he picked up a loose ball but he drove at the Hoops' box before his shot from 25 yards beat Fraser Forster and went in off the post.
Celtic responded positively and on the half-hour mark they were level when Izaguirre's deflected cross was flicked on by Samaras and Ledley stole in to head past Alexander from inside the six-yard box.
It was breathtaking drama and Rangers' fans thought their team had been awarded a penalty a couple of minutes later when Jelavic went down in the box as he went past Rogne in the box.
There was some confusion as referee Thomson appeared to immediately point to the spot but then gave a free-kick to the Parkhead men and booked the Ibrox striker, presumably for simulation.
Rangers started the second half strongly with Forster forced to push a Steven Whittaker drive past the post for a corner while skipper Davie Weir and Jelavic both had efforts which failed to hit the target.
Amid all that, Papac almost guided a Commons cut back past Alexander, the Ibrox defender happy to see the ball fly wide for a corner which came to nothing.
On the hour mark there was another Rangers penalty claim when Wilson appeared to use an arm to take the ball away from Maurice Edu as Celtic defended their box but again Thomson shook his head to the anger of the Light Blues' support.
Ki Sung-yueng replaced the injured Brown in the 65th minute and then Loovens - the sarcastic cheers of the Rangers fans - came on for Rogne but meanwhile play continued to race from end to end.
Kyle Hutton replaced Madjid Bougherra with eight minutes remaining and with the game looking to be heading for extra-time.
Moments later, following a Celtic corner, Ledley had attempt from 10 yards which was gratefully smothered by Alexander.
Then, after Hutton had bundled Izaguirre to the ground 30 yards out, Alexander made a comfortable save from Ki's free-kick.
Weiss replaced Lafferty just before the game moved in to extra-time.
Celtic looked fresher at the end of the 90 minutes but five minutes in to extra-time Forster made a great save from a Jelavic free-kick, re-adjusting himself at the last second to flick the ball over the bar.
However, two minutes later the Croatian put Rangers ahead again when he out-paced Mulgrew to latch on to a quickly-taken Weiss free-kick and his shot trundled past Forster and hit the post before rolling along the goalline and spinning into the net as Izaguirre tried in vain to clear.
Paddy McCourt came on for Commons but in the 110th minute Rangers had the chance to seal the win in a breakaway but Jelavic was pulled down at the edge of the box by Mulgrew, who picked up a yellow card.
El-Hadji Diouf replaced Jelavic and set up Weiss with minutes remaining on another Rangers breakaway but his shot was blocked by Forster.
Then, Whittaker blocked a goal-bound shot by Gary Hooper to conceded a corner before Izaguirre was shown a straight red for blocking Weiss as the Ibrox substitute made to run through to an open goal, with Forster up in attack to add his weight to the Hoops' threat at a corner.
It was the last incident of an incident-packed game which ended with Rangers picking up the first trophy of the season.