There was little between the teams in the first half, which saw Kelly score from the spot in the 26th minute after Gorgie defender Andy Webster had fouled James Dayton.
However, the Tynecastle men laid siege to the Killie box at times in the second period only to be denied by dogged defending and some fine saves from keeper Cammy Bell.
Hearts will wonder how they managed to leave Ayrshire with nothing for their efforts but that will worry Killie little, as they leapfrog the Edinburgh men to move into sixth place in the table.
In the 15th minute, Killie striker Paul Heffernan had the ball in the Jambos net but was ruled offside, and moments later Gorgie defender Ryan McGowan did well to thwart Dayton as he tried to get on to the end of a Cillian Sheridan cross from the right.
At the other end, Ryan Stevenson's deflected shot was parried by Bell and defender Michael Nelson beat Sutton to the loose ball, but the Tynecastle striker was flagged offside.
However, it was the home side's tricky wide-man, Dayton, who earned Kilmarnock a penalty when he was tripped in the Hearts box by Webster and Kelly confidently slammed the ball low past Jamie MacDonald.
Hearts began to play with more urgency and in the 33rd minute, striker Gordon Smith drove a wonderful volley past Bell from 20 yards but referee Steven McLean had already blown for a foul in the build-up.
After Dayton fired wide from the edge of the box after cutting in from the left, Rugby Park midfielder Borja Perez limped off to be replaced by Lee Johnson before Smith headed an Andrew Driver cross wide of the near post from eight yards.
Three minutes after the restart, Bell had to stretch high to his left to push Driver's goalbound drive from 20 yards behind for a corner, which came to nothing but which signalled an intense spell of Gorgie pressure.
As the game became a little tetchy, Dayton forced MacDonald into a decent save with a drive from the edge of the box following a Killie break.
However, Hearts remained on top and just after the hour mark a powerful volley from Mehdi Taouil could only be parried by Bell but Nelson calmly headed the loose ball back to his keeper.
In the 73rd minute Callum Paterson had a great chance when a slip by Rugby Park stopper Momo Sissoko gave him possession in the centre circle.
The Jambos substitute knocked the ball past Nelson to race through one-on-one with Bell but the Killie keeper was equal to what was an unconvincing shot.
In the 86th minute, Hearts captain Marius Zaliukas was shown a yellow for tripping Heffernan as the Killie striker chased the ball towards the Hearts penalty area, a decision which the home fans thought far too lenient, with Sheridan sending the resultant free-kick from 25 yards wide of the far post.
The visitors kept going through the four minutes of added time but eventually ran out of time and ideas, leaving the Killie fans and players to celebrate a welcome win.