In a match full of incident, Kingston slotted home at the death from Andrew Driver's cutback - the winger set up all of the hosts' goals - to consign the batting 10-man Highlanders to an eighth straight loss.
Despite being without a boss after Craig Brewster's sacking, the rock-bottom visitors were the better team in Gorgie before David Obua won a high ball battle to nod home his first Hearts goal towards half-time.
But Inverness, with first-team coach John Docherty in the dugout, grabbed a leveller with a maiden goal for Latvian defender Pavels Mihadjuks.
Michael Stewart saw a penalty saved before Roy McBain was dismissed and Obua restored the lead.
Dougie Imrie's stunning 88th-minute free-kick look to have rescued a point, only for Kingston to immediately net and secure a third straight victory for Hearts.
Both sides adopted cautious line-ups, with the recalled Garry Wood ploughing a lone furrow for the visitors and Christian Nade doing likewise for Hearts.
Michael Stewart - his knee heavily bandaged - registered the first attempt of note after 15 minutes, but he skewed well wide from 20 yards after a loose ball had reached him.
Inverness were forced into an early change when the recalled Thierry Gathuessi, who had picked up a knock in the far corner, was replaced by David Proctor.
The Highlanders soon offered muted claims for a penalty when Robbie Neilson and Imrie tangled shoulder to shoulder, but referee Steve Conroy waved away the appeals.
Imrie had the best opportunity of the game so far after 27 minutes.
Don Cowie's free-kick from the left arced to the back post where the unmarked midfielder somehow headed into the side-netting with the goal at his mercy.
Inverness were giving a decent account of themselves and again carved out a fine chance in the 35th-minute.
Iain Vigurs this time found space in the penalty area and his cross-cum-shot was deflected just past the right-hand post by the unmarked Wood, with Jamie MacDonald stranded.
Despite struggling to penetrate, the hosts made the breakthrough six minutes from half-time.
Driver's looping cross from the left saw Obua and Caley keeper Ryan Esson contest a high 50-50 ball yards from goal and the Ugandan came out on top to nod just over the line.
Esson redeemed himself after half-time, pushing away Driver's volley for a corner following a flowing move, before Christos Karipidis headed wide at the back post from Bruno Aguiar's cross.
The one-way traffic suggested Inverness were in for a tough second half, but they continued to offer a goalscoring threat and Mihadjuks glanced wide from a Cowie corner.
Hearts failed to heed the warning and paid the price after 56 minutes.
Cowie this time whipped in a free-kick from the left and Mihadjuks, one of Brewster's last signings earlier this month, powerfully headed into the middle of the goal past a helpless MacDonald.
Caley Thistle suddenly sensed an upset and and Ross Tokely should have done better than blaze over from 15 yards with Hearts caught sleeping from a corner.
Nade sought to turn the tide but Esson was equal to his low drive shortly after the hour, before the custodian also got behind a Stewart shot from distance.
Controversy then raged from the 68th minute. Driver's run to the byline was ended under a challenge from McBain and Conroy pointed to the penalty spot, despite fierce Inverness protests.
After a nervous wait for spot-kick taker Stewart as McBain was yellow carded, Esson produced a flying save to his right to keep the scores level.
But the drama was not yet over. McBain got himself in trouble again and, after consulting his fourth official, Conroy handed the defender a straight red card.
Amid the mayhem, Hearts sought to immediately take advantage but Esson tipped over Karipidis' header.
With a man advantage, the hosts were sure to find openings and the energetic Driver was again the architect for Obua to restore their lead in the 79th minute.
The young winger reached the byline on right and crossed brilliantly for his team-mate to head in at the far post.
Inverness looked finally beaten, but they stunned Tynecastle when Imrie curled in a superb free-kick just two minutes from the end.
Incredibly, though, Hearts were not to be denied and Driver raced straight up the park to cut the ball back for Kingston to net from close range and spark delirium.