Only three points would do for the Cumbrians as they relied on Northampton's result at Leeds for League One survival, but on a tense afternoon at Brunton Park they came up with their first win in 11 games.
Player-coach Kavanagh gave Carlisle a flying start with a superb right-footed strike from 28 yards after Joe Anyinsah had teed up the shooting chance.
Carlisle were full of industry and Lewis Neal should have extended their lead when Anyinsah's pass found him in the clear but the winger snatched at his shot which flew wildly over the bar.
The Cumbrians grabbed their second goal four minutes after the interval thanks to another spectacular effort, this time from skipper Thirlwell.
Kavanagh was lucky not to deflect James Henry's right-wing cross past his own keeper, before Gary Alexander also left Williams beaten with a header from six yards which came back off the underside of the crossbar.
League One play-off semi-final fixtures: First legs: Scunthorpe v MK Dons (Friday 8 May), Millwall v Leeds (Saturday 9 May, 12.45pm).
Second legs: Leeds v Millwall (Thursday 14 May), MK Dons v Scunthorpe (Friday 15 May).