Hearts had come back from James Fowler's brilliant opener 10 minutes into the second half when Rudi Skacel and Ryan Stevenson netted.
But on-loan Everton striker Agard volleyed home in the 86th minute as Kilmarnock capitalised on some sustained pressure.
The thrilling finale made up for a scrappy and bad-tempered first half and saved Killie caretaker manager Kenny Shiels from a third successive defeat, although he remains without a win in five games.
Former Hearts manager Csaba Laszlo, who has expressed an interest in the Rugby Park post, watched from the main stand as Hearts moved eight points clear of Dundee United, who play at Celtic Park on Sunday.
Hearts again had to reshuffle at the back with Andy Webster missing out with a groin injury and Ismael Bouzid suspended, although Marius Zaliukas returned from a ban as Adrian Mrowiec stepped in at centre-back.
Mohamadou Sissoko returned from a groin injury for Kilmarnock with Willy Aubameyang starting up front.
Hearts threatened first when Cammy Bell parried a powerful long-range strike from Ian Black and Eggert Jonsson headed over from inside the six-yard box after Ruben Palazuelos had flicked on Craig Thomson's corner.
Jamie MacDonald, back in goal for Marian Kello, saved Hearts in the 14th minute with a brilliant one-handed stop to push Alexei Eremenko's 30-yard free-kick on to the bar.
Skacel's free-kick delivery from the right wing almost crept inside Bell's far post but the game became a bad-tempered affair until the break with stoppages adversely affecting the entertainment.
Stevenson and Sissoko were booked for bad challenges before Thomson and Tim Clancy were cautioned after the Kilmarnock full-back squared up to his opposite number following a robust challenge after the ball had gone out of play.
Black was then booked for persistent fouling by Euan Norris before David Templeton came close with an 18-yard drive following an enterprising run.
Kilmarnock started the second half positively and Craig Bryson scuffed a shot at MacDonald, who soon pushed Manuel Pascali's header wide following Jamie Hamill's free-kick.
Eremenko was booked for diving in the box before the hosts took the lead in the 55th minute when Liam Kelly laid the ball back to Fowler to strike a rising shot into the corner of the net from 25 yards.
Hearts immediately threatened from two set-pieces but were struggling to get in behind their hosts before Skacel levelled with a scrappy goal in the 66th minute.
Palazuelos swung in a cross from the left towards Stevenson, who could not make any significant contact. But the ball bounced over the head of Bell and on to the bar before Skacel slid in to net the rebound.
Black soon had a good chance when Stevenson's backheel opened up the Killie defence but the midfielder's left-footed shot cleared the bar from 16 yards.
Agard replaced Aubameyang but he had not touched the ball before Hearts went ahead in the 76th minute when Stevenson headed Thomson's inswinging corner into the corner of the net from six yards.
Skacel shot over after Stephen Elliott's inside pass before Kilmarnock rediscovered their attacking momentum.
Da Silva shot into the side netting from 25 yards before substitute James Dayton forced a diving save from MacDonald from his inswinging free-kick.
Agard missed a great chance to equalise when he back-headed Eremenko's cross over the bar from four yards.
But he soon atoned when he swivelled to volley into the top corner from 12 yards after positive play from Dayton and Eremenko. Hamill earned the game's seventh yellow card before Dayton and Da Silva both had chances to win it in the closing moments when they were played through, but MacDonald saved both times.