Lee Hughes looked to have popped up with the winning goal for the League One side in the eighth minute of extra-time when he placed a right-footed shot beyond the reach of Paul Smith and into the bottom-right corner.
However, Wes Morgan levelled in the 122nd minute with a sensational 25-yard shot.
Hughes had the chance to win the subsequent shootout for his side following George Boateng's miss, but the striker lifted his penalty over the bar and Neal Bishop swiftly followed suit to hand Forest victory.
Former Forest striker Craig Westcarr had earlier levelled for County with 14 minutes of normal time remaining, after goals from Lewis McGugan and Robbie Findley put the hosts in front following Mike Edwards' early opener.
The League One side made a bright start and, after Bishop had struck the bar, took the lead in the 16th minute when Edwards converted Alan Sheehan's right-wing cross.
However, Forest pulled level in stunning fashion on the half-hour as McGugan curled home a wonderful 30-yard free-kick off the underside of the bar.
Stuart Nelson reacted well to tip Andy Reid's header around the post as Forest pushed for a second.
Yet it was the Magpies who were on top at the beginning of the second half and Smith had to make a fine save to deny Ricky Ravenhill.
The momentum soon shifted again and Findley converted Reid's left-wing cross in the 56th minute to put the hosts ahead, although Nelson may feel he should have kept out the shot.
Nelson did deny the American soon after and Luke Chambers fired over from a McGugan cross before County equalised with 14 minutes remaining.
Forest were guilty of failing to clear their lines and substitute Westcarr was on hand to provide a clinical finish.
County duly ended the 90 minutes as they had started it, as the stronger side, and after eight minutes of extra-time they took the lead for a second time through Hughes' superb strike.
Morgan, Jonathan Greening, Marcus Tudgay and Joe Garner all had opportunities to take the game to penalties, but failed to hit the target.
However, Morgan then came up trumps in the second minute of injury-time as he crashed home from long range.
Greening hit the post with Forest's first penalty, but Hamza Bencherif blazed over with County's third spot-kick.
After Boateng and Hughes had failed from 12 yards, Luke Chambers kept his cool for the home side before Bishop's costly miss.