Goals from Josh Magennis and Tope Obadeyi put them on top but they had to hold on for dear life after conceding a goal to substitute Liam Boyce inside the final 10 minutes.
County almost grabbed the lead in the second minute when Yoann Arquin fired in a shot from 35 yards that was deflected over the bar by defender Chris Chantler.
Graham Carey’s effort went narrowly over the bar and he had a free-kick from distance claimed by Killie keeper Craig Samson at the second attempt.
But despite the home side’s pressure, it was Kilmarnock who grabbed the lead six minutes before half-time, Magennis climbing well to head home a cross from Craig Slater.
Magennis headed wide after the interval as Killie pressed for a second, and it came on 56 when Ross Barbour clipped the ball into the back post area and Obadeyi pounced to beat Antonio Reguero.
Slater tried his luck from 30 yards and was close to making it 3-0 before Uros Celcer came close to opening the game up with a left-foot effort saved by Samson.
The home side kept pushing to get back into the match and Carey made a darting run into the box only to be denied by a last ditch tackle by Magennis.
But County were thrown a lifeline in the 81st minute when Boyce got on the end of a deep cross from Cardle to head in from close range.
However, despite a late flurry of activity towards the Killie goal the visitors held out for their first win of the campaign.