Goals from Stewart, on early for the injured Tam McManus, and Carl Finnigan had given the Bairns the interval advantage over a Hearts side who had right-back Craig Thomson sent off by referee Brian Winter in the 26th minute.
Thomson was ordered off for kicking out at Finnigan after the pair clashed on the touchline in the Co-operative Insurance Cup third-round tie at the Falkirk Stadium.
In a breathtaking second-half fightback, Kevin Kyle reduced the deficit from the penalty spot in the 66th minute and Suso Santana levelled before Kyle headed the visitors into the lead.
But with 10 minutes remaining Ryan Flynn scored for the Irn-Bru First Division side and with seconds of the three added minutes remaining, Stewart popped up in the six-yard box to knock in the winner.
It had not looked good for Falkirk when they had to make a change in the fourth minute when former Hibs striker McManus hobbled off following a touchline challenge with Santana, to be replaced by Stewart.
However, the disruption did not affect the home side who created the first chance in the sixth minute when Jack Compton battled well near the touchline to cut the ball back for Finnigan whose shot from 12 yards sped across the goal and past the far post.
The visitors failed to heed the warning and with their defence wide open again, Compton played in Stewart who drove under goalkeeper Marian Kello for the opener.
The game became more tetchy after Thomson was sent off for kicking out at Finnigan as they both lay prostrate, after referee Winter took instruction from his assistant Stuart MacAulay and his fourth official Eddie Smith.
Hearts boss Jim Jefferies waited a few minutes before replacing striker Elliot with Adrian Mrowiec.
But five minutes from the break, Stewart raced on to a pass from Kallum Higginbotham, pulled the ball back from the by-line and the unmarked Finnigan sidefooted past Kello.
Hearts came out in determined mood at the start of the second half but within two minutes Falkirk almost made it 3-0 on the break when Finnigan's angled drive from 16 yards was straight at Kello.
An increasingly frenzied cup tie swung from end to end and just after the hour mark, with huge gaps in the Hearts defence again, Stewart drove over the bar from the edge of the box after taking a pass from Compton.
David Templeton went close for Hearts when his drive zipped past the post, before Winter, at the behest of his fourth official Smith, had to warn the Hearts bench to calm down.
But they had reason to get excited again soon after when Kyle converted the spot-kick awarded when Bairns skipper Burton O'Brien felled Santana inside the box.
There was more drama when Santana drove in from just inside the packed Bairns penalty area after the home side had failed to clear Ryan Stevenson's corner.
To the delight of the travelling fans behind the goal, Kyle then rose to head a Stevenson free-kick in off the crossbar to give the visitors the lead before Flynn drove in a wonderful left-footed shot to draw the teams level again.
The game looked certain to be heading for extra-time but Compton's corner from the right landed at Stewart who knocked in from close range to send the Bairns fans into ecstasy.