Roberto Mancini's men needed to win to have any chance of qualifying for the Europa League in third place in the group but the Premier League champions could find no way through in Germany and were finished off by Julian Schieber's second-half stike.
In a goalless first period of few chances, Edin Dzeko had the best City effort but saw his shot from 25 yards saved by Roman Weidenfeller diving low to his right.
At the other end, Joe Hart made good saves to his left to deny first Marco Reus and then Ivan Perisic.
In the second half, Dzeko headed straight at the keeper with a close range effort soon after kick-off.
Mats Hummels just failed to convert a free-kick from Marcel Schmelzer before Hart had to push over a Perisic volley and save to his right to deny Kevin Grosskreutz.
The hosts deservedly went in front on 57 minutes when Schieber slotted home from six yards after a fantastic cross from substitute Jakub Blaszczykowski.
Soon afterwards, Schieber was played through on goal and he tried to chip the on-rushing Hart but the City keeper caught the ball.
Mancini threw on Sergio Aguero and Mario Balotelli in search of the two goals now needed but it was Hart who had to make another great save to deny substitute Robert Lewandowski from close range.
Aguero tried an effort which Weidenfeller saved before the Argentinean played Balotelli in on goal but he was given offside and saw his shot saved.
Balotelli was booked in injury time for throwing the ball to the ground in frustration which summed up a miserable night for City.
The English champions tamely dropped out of Europe without notching a win in the group - the English team not to win a single game in the Champions League group phase.
In time, they may find the lack of European action assists them in their title defence, but they must get the disappointment out of the system quickly before Sunday's derby against Manchester United.