Peter Vincenti scored the winner from the spot, moments after the visitors thought they should have had a penalty of their own.
Both sides had good chances but couldn’t alter the scoreline as Britt Assombalonga was twice denied by Josh Lillis in the Rochdale goal, while the hosts had chances to double their lead.
But the League One outfit held on, despite huge pressure in the second half, to take a fine victory over the Championship side, who have now not won in seven matches.
Chris Burke looked lively early on and produced a fine run before being brought down on the edge of the box, and David Vaughan struck the resulting free kick at the goalkeeper.
Burke then felt he should have had a penalty as he raced into the area and appeared to be tripped, but the referee gave nothing.
Moments later, their fury was compounded as the hosts earned a spot kick at the other end. Joe Rafferty was felled in the area and Peter Vincenti stepped up to give the hosts the lead early on.
Both sides had chances as Britt Assombalonga – who had five goals in his previous four FA Cup appearances – fired at the goalkeeper before Michail headed over from six yards, while Jamie Allen nearly doubled the hosts’ lead before the break but for a fine save from De Vries.
Stuart Pearce’s half-time words clearly had an effect as Forest improved greatly after the break, with Lars Veldwijk heading Burke’s cross just wide of the post.
Assombalonga then tested Josh Lillis once again, racing down the inside left channel before forcing a good low save from the Rochdale goalkeeper.
The ever-lively Matt Done almost made it 2-0, turning on the edge of the box and drilling an effort just wide of the upright before Assombalonga headed another chance over the bar for the visitors.
Forest pressed and pressed as the half went on but they were unable to convert it into an equaliser, and the hosts held on for a famous victory to make it into the fourth round.