Danny Grainger and Rudi Skacel were both denied by Ayr goalkeeper Kevin Cuthbert in the shoot-out while Mark Roberts coolly slotted home the winning spot-kick for the First Division side, who eliminated Inverness in the previous round.
In normal play Gareth Wardlaw equalised to cancel out Scott Robinson's 48th-minute opener for Hearts and send the tie to extra-time.
Paulo Sergio made eight changes to the Hearts side that triumphed against St Mirren in the Clydesdale Bank Premier League on Saturday, with former Ayr midfielder Ryan Stevenson one of those who missed out.
The Edinburgh side started the game the brighter and their first chance came when John Sutton rose to connect with Ryan McGowan's cross, only for Cuthbert to produce an excellent low stop.
David Templeton drove an effort into the side-netting four minutes later before the hosts offered their first attempt on goal, Michael Moffat firing high over the crossbar after Michael McGowan's corner was only partially cleared by the Hearts defence.
Moffat was involved again minutes later when he fired in a dangerous delivery for Wardlaw, who headed inches over the top with Janos Balogh in the Hearts goal in no man's land.
Balogh looked nervous again in the 28th minute when he came and failed to collect McGowan's fizzing corner but nobody in an Ayr shirt was able to capitalise.
Hearts were quick out of the traps at the beginning of the second half and they grabbed the lead on 48 minutes.
Grainger's corner was headed only as far as Robinson, who drove the ball low beyond Cuthbert and into the bottom-left corner of the goal.
Ayr almost hit back immediately when McGowan's cross evaded everyone in the penalty area but Eddie Malone was unable to convert at the back post.
Grainger's set-pieces continued to cause the Ayr defence problems and on the hour mark he looked to have helped the visitors double their advantage after Eggert Jonsson bundled the ball home, only for referee Ian Brines to disallow it after deliberating with assistant referee Tom Murphy, who ruled the Icelandic defender had punched the ball into the net.
That decision was to prove even more significant just two minutes later when Marius Zaliukas slipped 20 yards from his own goal allowing Wardlaw to drive through on goal and slide a neat finish beyond Balogh.
Hearts began to pressure their First Division opponents and created chances through Templeton and Sutton but their opponents comfortably soaked up the pressure.
Sergio replaced Templeton with Mehdi Taouil on 75 minutes and replaced Sutton with Gordon Smith in a bid to clinch the game but it was Ayr who almost snatched victory when McGowan's wind-assisted corner forced Balogh into a smart save.
Neither team was able to create any further openings in the closing stages meaning a hard-fought encounter played in difficult conditions moved into 30 minutes of extra-time.
Ayr boss Brian Reid made an early change when play resumed - replaced scorer Wardlaw with Mark Roberts.
Hearts appeared to be running low on ideas and with the exception of Grainger's long range effort, Cuthbert was barely tested during the opening period.
The visitors raised the tempo in the opening stages of the second period forcing multiple corners and being denied by a number of match-saving blocks by Cuthbert, the pick of them coming from Ross Robertson after a stinging strike from Skacel.
Taking the kicks in front of the visiting support, Grainger and Skacel saw their efforts kept out by Cuthbert while Moffat, Jamie McKernan, Alan Trouten and Roberts all successfully converted from the spot for Ayr.
