Mikele Leigertwood's first Royals goal had put the Championship side in control of Saturday's fourth-round tie but Darius Charles appeared to have earned the hosts a replay with a 78th-minute goal.
But Long grabbed his ninth goal in nine games with just three minutes left to put the Royals into round five.
The League Two side beat Newcastle in round three but failed to reach those heights as the Royals made it through.
Michael Bostwick scored against Newcastle and had the first effort on goal with a 25-yard strike that was straight at Adam Federici in the Reading goal.
Liverpool were beaten as Reading reached the quarter-finals last season and the Royals soon found their stride again.
Veteran full-back Andy Griffin waltzed into the area before pulling a cross across the face of goal and in-form striker Long then headed wide from a corner.
But the Championship side did take the lead through Leigertwood in the 23rd minute.
The on-loan QPR man dribbled through the middle and beat two challenges before striking a left-footed effort past Chris Day and into the corner for his first goal since last April.
Long then went tumbling inside the box but was not given a penalty before the home side came close.
Left-back Scott Laird was attacked by a fan after the win over Newcastle but almost landed a blow of his own with a volley that flew just wide of Federici's post.
Ian Harte threatened with a free-kick over the top as the half came to a close.
Simon Church turned sharply to shoot over the top at the start of the second period as Brian McDermott's men looked for a killer second goal.
Stevenage made a tactical switch with left-back Charles moving up front but the Royals' backline stood firm.
Reading should have made the game safe just after the hour mark.
Jobi McAnuff beat his man down the right and delivered a dangerous low cross to the far post. Hal Robson-Kanu was in space but hesitated to take a poor touch instead of hitting a first-time shot and Day was able to smother the ball at his feet.
It was a major let-off for the underdogs and they almost punished Reading moments later when Chris Beardsley thumped a header over the top.
Charles was proving a threat however and he levelled with 18 minutes left.
Stevenage moved the ball quickly out onto the right-hand side where Charles had found space. He ran in on goal and cut inside Matthew Mills before curling a fine effort wide of Federici from the corner of the penalty area which found the corner.
Charles sent another effort just wide before limping off with an injury as the tie entered the last 10 minutes.
And Long sealed the win for the visitors in the 87th minute when McAnuff again got to the byline before standing up a cross which Long volleyed in from close range.