Jamal Campbell-Ryce looked set to be the hero for County as he converted a 71st-minute penalty after the usually reliable Jeff Hughes had missed from the spot minutes earlier in the League One clash.
But Stevenage levelled with a penalty of their own in the 83rd minute when Lucas Akins converted after County's Alan Sheehan had deliberately handballed and was subsequently sent off.
It looked as if the home side would take the lead after 52 minutes when Jeff Hughes was upended by Akins before taking the penalty himself, only to hit the post.
But Campbell-Ryce made no mistake when he had the chance from the spot after Mark Roberts had fouled Francois Zoko.
Akins's spot-kick and Sheehan's sending off gave the visitors the impetus and they took advantage of that when substitute Robin Shroot won it in the 90th minute, finishing well into the bottom corner.