The 48-year-old has paid for a run of two points from four games which means the Scots - bottom of Group A - have little chance of reaching their first finals in 16 years.
A seven-man Scottish Football Association panel has been considering Levein's future since late last week, and the former Hearts and Dundee United manager will not now be in charge for this month's game against Luxembourg.
That friendly, against opponents ranked 142nd by FIFA, would represent a gentle start if a seventh permanent manager since the turn of the century is found in time.
Gordon Strachan is a clear favourite with bookmakers Sky Bet, with former bosses Walter Smith and Alex McLeish also prominent in the betting.
Levein, who succeeded George Burley in December 2009, leaves with a record of 10 wins, nine defeats and five draws from his 24 games.
Several senior players had spoken out in support of the manager before his departure, which was met with disbelief by midfielder Charlie Adam.
He tweeted: "On my way back from training to hear Craig Levein has gone. Unbelievable. Can't believe the Scottish papers got what they wanted."