Championship side Forest sacked Billy Davies on Monday, having dropped to seventh in the standings after a bad run of results.
And the experienced Warnock, who has guided several teams to promotion during his long career, was believed to be in pole position to take charge until the end of the season, having reportedly met with owner and chairman Fawaz Al Hasawi on Sunday.
However, Forest will now have to look elsewhere following Warnock's announcement.
"I did not feel the conditions of the job were right for me to manage. I wish them well for the rest of the season," Warnock said.
The 65-year-old's decision is sure to concern Forest fans, who on Monday afternoon were told to be patient by chairman and owner Fawaz Al Hasawi.
Al Hasawi has revealed that academy manager Gary Brazil will take charge of first-team affairs until a permanent appointment is made, starting with Tuesday night's game against Charlton at the City Ground.
Al Hasawi told Sky Sports on Monday: "Me and Billy will always be friends and I don't have anything against him. This is business and sometimes it doesn't work.
"I have spoken to the players today and Gary Brazil held a good training session today, I was watching.
"The players are looking forward to the game tomorrow (at home to Charlton) and we want to make the fans happy tomorrow. I really feel sorry for the fans, they deserve better."
Asked who would be taking charge, Al Hasawi said: "We had a chat with Neil Warnock's agent, but nothing has happened at this moment, the job is open for other managers too.
"Maybe we will carry on with Gary Brazil, maybe we sign a new manager after tomorrow, maybe we will carry on until the end of the season.
"I don't want to rush. There is no plan at the minute, I want to bring the right manager to Nottingham."