Ayr veteran Mark Roberts' impressive footwork and perfect finish in the 19th minute decided the fourth-round replay and left Hibernian fans with little to look forward to for the rest of the campaign other than retaining their SPL status.
Hibs found Ayr goalkeeper David Crawford in excellent shot-stopping form but they struggled to consistently create chances and are now without victory in nine games.
Calderwood had left out the five midfielders who started Saturday's 3-0 home defeat by Celtic with Colin Nish and Darryl Duffy introduced up front and Riordan moving to left midfield.
But the winning formula continues to elude the Hibs manager, who has tasted victory in just two out of 15 games since being brought in to an arrest a slump that now stretches back almost a year.
Calderwood began his early football education as a schoolboy with Ayr but he was taunted with chants of "you're getting sacked in the morning" by some of the home fans, who formed the vast majority of the 3,826 crowd.
The Irn-Bru Second Division leaders matched Hibs from the start, creating the first chance when Chris Smith blazed well over after a corner. The visitors soon responded with Kevin McBride's weak effort cleared off the line by Eddie Malone after a mix-up between the defender and Crawford.
Lone striker Roberts threaded Andy Rodgers in down the inside right channel and the Ayr man appeared to be held by Paul Hanlon before recovering to stab a shot past Brown but beyond the far post.
The same two players combined for the goal after Hibs were caught on the break following a long spell of possession. Rodgers returned the ball to Roberts who took it down just inside the box, but there appeared to be limited danger as Ian Murray and Hanlon blocked his way to goal.
However, the former Kilmarnock striker moved the ball from foot to foot until he had time and space to shoot, and guided the ball into the bottom corner.
Hibs had an instant chance to level when Duffy's inside ball found John Rankin, who charged into the area unchallenged but saw his shot blocked by the foot of Crawford.
Ayr sat back but Hibs rarely got down the side of the organised home defence and the SPL side only created one more clear chance before the break.
The hosts were caught square and Duffy burst towards goal with Crawford rooted to his line. But the Ayr goalkeeper parried Duffy's angled effort and the home defence blocked Riordan's follow-up.
The underdogs started the second half brightly but right-back Jonathan Tiffoney almost gifted Hibs a 57th-minute equaliser when he tried and failed to take the ball round Riordan inside the box after Nish had won a long free-kick in the air.
Riordan immediately curled the ball towards the far corner but Crawford threw himself to his left to push wide. Ayr soon had another narrow escape when Willie Easton nodded Hanlon's header off the line following a corner.
Ayr responded and both Roberts and Rodgers came close to converting Tiffoney's low cross.
Merouane Zemmama replaced Duffy in the 65th minute to take up a position just behind the front two, which now featured Riordan.
Miller soon saw a shot deflected just wide after a lay-off from Nish, whose header from the resulting corner was parried wide by Crawford.
But Hibs failed to keep up the pressure and Brown almost spilled Scott McLaughlin's free-kick at the other end.
The home players appeared to find extra energy as they chased every ball before inevitably being pegged back in the final 10 minutes.
But Crawford held Riordan's powerful strike and the home defence charged down every other effort on goal before the final whistle sparked a pitch invasion from the jubilant home fans.