The resilient visitors were applauded off by the home fans and their 1200 travelling supporters after almost forcing a replay against their SPL hosts.
Resolute defending kept City in the tie up until the second minute of stoppage time when substitute Dani Sanchez broke the deadlock and top scorer Adam Rooney settled things two minutes later.
The Morayshire visitors, who are fourth in the Scottish Third Division, can return home 40 miles east proud of their performance in the first competitive meeting of the sides but Inverness proved their superiority in the end.
Caley manager Terry Butcher made two changes to his starting line up following last Sunday's 1-0 defeat at St Johnstone with Northern Ireland goalkeeper Jonny Tuffey replacing Ryan Esson, while club captain Richie Foran was restored ahead of Gil Blumenshtein, who was left out of the squad.
Elgin boss Ross Jack made one change to the side who drew 1-1 at Queen's Park last time out on 18th December with Craig Gunn in for David Niven.
A cagey opening 15 minutes was sparked to life when Inverness full back Graeme Shinnie found room to threaten.
The 19-year-old shot inches wide from the edge of the box after being played through by Eric Odhiambo.
Moments later Odhiambo squandered a great chance when he powered a free header wide from a Nick Ross cross.
The game then fell back into a midfield battle with neither side showing any cutting edge.
Gunn had a good turn of pace against Grant Munro but ran the ball out of play and Rooney might have scored if he had connected with a cut back from Foran.
Caley Thistle started the second half brightly with Rooney almost turning provider when his cross was half-cleared as far as Foran, who volleyed over.
Neither goalkeeper was tested until the 49th minute when Elgin's Peter Donnelly palmed a Rooney shot wide.
The Irish attacker had been played in by Shinnie but his angled effort from inside the area was thwarted.
Moments later Odhiambo blew another chance when his shot went wide after being found by Lee Cox.
Rooney steered a header wide just before the hour when he met Stuart Duff's cross as Inverness appeared to be taking control.
But against the run of play the visitors produced their best chance when Jason Crooks charged through and forced Tuffey to punch away his shot, which dropped to Craig Frizzell whose effort rippled the side netting.
Donnelly made a fine diving save from Inverness substitute Shane Sutherland with 20 minutes to go after he was played in by Ross.
The stopper was equal to an overhead kick by Foran seconds later when Ross created yet another opportunity.
Ross was again at the centre of things in the closing stages when he found Sutherland on the edge of the area and his low blast was touched just wide of the target by Donnelly.
Crooks was in the right place at the right time to block Rooney's goalbound header following a corner from Sanchez which was guided back across goal by Munro.
But it was heartbreak for City when the Spaniard hammered home from the edge of the box in the second minute of stoppage time.
Two minutes later Ross crossed to Sanchez who nodded it to Rooney who fired home from close range.