Reds boss Billy Davies altered his entire starting XI from the one that had claimed a creditable 1-1 draw at Watford but eventually saw his reshuffled side provide a demonstration of their strength in depth, as an extra-time goal from academy product Jamaal Lascelles steered them to victory.
A 58th-minute header from Matt Derbyshire looked to have settled the tie but Liam Feeney's drive, possibly deflected by substitute Andy Keogh, levelled matters for the Lions with just four minutes left.
And it needed Lascelles' 94th-minute header, after Greg Halford had headed a Jamie Paterson corner back across goal, to win the game for Forest.
Millwall carved out the first clear opportunities as Lee Martin launched himself into a twisting run down the right before setting up Martyn Woolford for a shot from the edge of the box, which Dorus de Vries saved adeptly low to his right.
Scott McDonald also tested the reactions of the former Swansea and Wolves keeper, who had to make an acrobatic save.
Forest had given a full debut to one of their nine summer signings, Algerian winger Djamel Abdoun, and the capture from Olympiacos initially looked the most likely to make something happen for the home side. A cross from him, to the far post, saw Lascelles arrive to head powerfully wide.
But Millwall were carving out the clearer opportunities on the break, with Nicky Bailey pushing forward to force De Vries into a third save of the first half with a low, skimming effort from 25 yards.
Dan Harding forced visiting keeper Stephen Bywater to make his first save of note in the 47th minute, with a crisp shot that was flying low towards the near post.
It was again Millwall who looked likely to break the deadlock, with McDonald flashing a shot wide when Woolford's persistence had carved out a clear opportunity from 12 yards.
But that proved to be a decisive moment as, after Abdoun had forced a Forest corner, the home side took the lead. A quick routine gave Paterson the chance to cross and he picked out Derbyshire, who glanced the ball home from a few yards out to give Forest a 58th-minute advantage.
Substitute Ishmael Miller also forced two smart saves from Bywater, amid a lively cameo. Miller's failure to capitalise was significant, however, as Millwall pulled level in the 86th minute, with Feeney drilling the ball low past De Vries as Keogh appeared to get the slightest of touches.
Substitute Dexter Blackstock looped a header over as Forest pushed for the lead a second time, in extra time, before Lascelles did restore their advantage.
De Vries had to produce another fine save to deny Woolford and Millwall but Forest held on.