The full-back's low strike in the 89th minute secured the points for Caley, giving them their first victory of the campaign.
The hosts took the lead in the 26th minute when Gregory Tade powered home a close-range header before Dean Shiels levelled with a 25-yard curling shot nine minutes later.
Inverness had made three changes to the side dumped out of the League Cup in midweek at Ayr.
Influential midfielder Jonny Hayes has been sidelined after suffering a dislocated shoulder, while goalkeeper Jonny Tuffey and forward Shane Sutherland were dropped to the bench, allowing Ryan Esson, Nick Ross and Tade to start.
Kilmarnock also made three changes, with Ryan O'Leary and Paul Heffernan injured and Gary Harkins starting on the bench, as defenders James Fowler and Zdenek Kroca and striker Ben Hutchinson started the game.
Inverness midfielder Nick Ross went close to opening the scoring with a thunderbolt on the quarter-hour mark.
His rasping 25-yard shot went just wide of the target after he was found by captain Richie Foran.
Inverness had started brightly as they hunted for their first home goal of the season.
Ross fired inches over after latching onto a Foran header from Aaron Doran's corner.
Tade forced the first save from Cameron Bell when he met a long ball from Ross Tokely but the Frenchman's shot lacked conviction.
Zdenek Kroca was the first to threaten for the visitors when he headed over a corner from Shiels.
Tade's first goal for Caley Thistle put the hosts in front.
The forward, who signed as a free agent after playing for Raith Rovers last season, put the Highlanders ahead with a powerful header from six yards.
The Frenchman evaded his marker to connect with a pinpoint cross from Foran - who was making his 100th appearance for the club.
The visitors responded superbly with the equaliser from Northern Ireland international Shiels, who curled a low effort beyond a diving Esson from 25 yards after he was found in space by Liam Kelly.
Kelly appeared to have been hit by an object thrown from the home fans while celebrating the goal.
Rory McKeown might have put the visitors in front but his rising 30-yard strike from a short James Dayton free-kick was just off target.
Hutchinson forced an acrobatic save from Esson after Dayton played him through, with Inverness struggling moments before the interval.
Tade's looping header might have restored the lead for Inverness within a minute of the restart but he directed it inches wide after being picked out by Shinnie's cross.
Doran dragged a long low shot wide before putting a free-kick on target as the home side showed their attacking intent.
Kilmarnock captain Manuel Pascali connected with a Shiels cross but headed wide as the game developed into an end-to-end affair.
Hutchinson's low free-kick from the edge of the area was saved well by Esson on the hour mark.
Seconds later the Kilmarnock forward beat Esson but was denied by the post after defender Chris Hogg had been caught in possession on the edge of the box.
Foran squandered a great chance to put Caley Thistle in front when he sent a diving header wide after being found unmarked by Doran eight yards out.
Under-strength Inverness suffered a double injury blow in the closing stages with Foran and Doran both going off for treatment.
But with two minutes to go Shinnie lifted the gloom with the winner.
The full-back's shot from 10 yards found the bottom corner after he was picked out by Ross.