Ross Draper netted the decisive penalty, coolly finding the bottom left-hand corner of the net after sending Jamie MacDonald the wrong way.
However, in a thrilling topsy-turvy contest, Hearts were just seconds away from reaching the final for a second straight year when Nick Ross popped up to fire home the equaliser from close range following a scramble in the box.
By this point, Inverness had been reduced to nine players, following the dismissals of Gary Warren and Josh Meekings.
Inverness took a deserved lead in the 54th minute when Greg Tansey latched on to a loose ball to fashion a blistering 23-yard strike past a diving and helpless MacDonald.
But the game was turned completely on its head during a three-minute spell which saw Caley Thistle reduced to 10 men followed by two Hearts goals.
First, Warren was harshly sent-off on 66 minutes after collecting a second yellow card, the latter for a foul on Sam Nicholson on the edge of the box.
And, from the resulting free-kick, Hearts were level when the ball was touched by Kevin McHattie to Jamie Hamill whose low shot took a hefty deflection off Draper and beyond Dean Brill into the bottom left-hand corner of the net.
Two minutes later shell-shocked Inverness fell behind but there were no complaints on this occasion as Hamill superbly curled a free-kick past Brill from 25 yards.
To make matters worse for Inverness, Meekings was sent-off in the closing moments for a desperate lunge on Scott Robinson.
After Ross had levelled matters in the fourth minute of injury time, Hearts substitute Paul McCallum missed the best chance of the overtime period, heading over from close range after 118 minutes.
And so to penalties where Graeme Shinnie’s opener for Inverness was saved, while McCallum also failed from 15 yards leaving the shoot-out scoreless.
Billy McKay – who had missed the best chance of a goalless first half – fired Inverness ahead before Dale Carrick converted Hearts’ second penalty.
Ross was on target for Inverness but Hamill had his effort blocked by Brill to keep Caley Thistle ahead.
Tansey then made it 3-1, before Robinson pulled one back for Hearts only for Draper to seal the victory.
This unlikeliest of triumphs for Inverness avenged their defeat by Hearts at the same stage of this competition last season when the match also went to penalties.
They now meet Aberdeen in the final next month at a venue yet to be decided.