Ian Vigurs netted the decisive spot-kick in a thrilling encounter as Paul Tosh and Paul Lunan both missed for the home side to see them crash out 4-2 on penalties.
That came after Caley, reduced to 10 men after an extra-time red card for Andy Barrowman, were twice ahead through Vigurs and Garry Wood only for Tosh and Barry Sellars to cancel out their strikes.
Barrowman hit the bar with a header with five minutes left of the added time and there was drama in the final moments with five players booked and one red-carded in a fiery incident.
Scott Gates and Phil McGuire clashed with both pushing each other to the deck.
All 22 players raced to the scene, with Gates, McGuire, Robbie Raeside, Ryan Esson and Barrowman yellow-carded, with the latter sent off for his second bookable offence.