Levein has paid the price for Scotland's poor start to their World Cup qualifying campaign, which has left them bottom of Group A with two points from four games.
Scotland U21 coach Billy Stark will take temporary charge of the team and his first task will be to name a squad for next Wednesday's friendly in Luxembourg.
An SFA statement read: "The Scottish FA tonight announces that Craig Levein has been relieved of his duties as Scotland national coach.
"The board of the Scottish FA has taken the decision primarily due to the disappointing results in the FIFA World Cup Brazil 2014 qualifiers."
The SFA have stressed that they will honour Levein's contract after he expressed a desire to remain in charge until the end of the World Cup campaign.
SFA chief executive Stewart Regan said: "We have not terminated the manager's contract. He will be paid his dues and his entitlement under his contract."
Levein's job had been under threat since defeats by Wales and Belgium last month and his overall competitive record - three wins from 12 matches - is among the worst of modern Scotland managers.
And a seven-man Scottish Football Association panel has finally decided to show the 48-year-old the door, having been considering his future since late last week.
However, they will come under fire for the timing of the decision, as the squad to face Luxembourg had been due to be named on Tuesday.
Focus will now turn to a permanent successor and former Celtic manager Gordon Strachan is a clear favourite with Sky Bet, while former bosses Walter Smith and Alex McLeish are also prominent in the betting.
Levein appeared to have the support of his players prior to his dismissal, several of them having spoken out on his behalf.
And midfielder Charlie Adam tweeted after hearing of the sacking: "On my way back from training to hear Craig Levein has gone. Unbelievable. Can't believe the Scottish papers got what they wanted."