Coming into the game with a one-goal advantage having won the first leg 3-2, Celtic were undone by Markus Rosenberg on 22 minutes and an own goal from Dedryck Boyata on 53 minutes, with both goals coming via corner kicks.
Celtic may feel aggrieved at a Nir Bitton goal just before half-time which was strangely disallowed, but apart from that they rarely threatened the Malmo goal.
Celtic last featured in the Champions League group stage in 2013-14 but lost to Maribor of Slovenia in last year’s play-offs.
The hosts had not conceded in their last nine European qualifying home games and started intent of making a statement. Roared on by their raucous home fans, they took the lead on 22 minutes.
Rosenberg, who missed the first game at Parkhead through suspension, took advantage of some lightweight Celtic marking at the near post to flick home a right-wing corner.
The visitors were struggling to deal with the pressurised situation but did have the ball in the net just before half-time but the referee’s whistle cut short the celebrations.
Bitton poked home inside the six yard box after a Malmo defender handled the ball as he attempted to head the ball clear. It looked a clear penalty and a goal but the referee deemed Celtic striker Leigh Griffiths had fouled the goalkeeper.
Ronny Deila decided to throw on Kris Commons at half-time to inject some quality into their play but their task became tougher just seven minutes into the half.
A corner from the left once again was defended abysmally by the Celtic defenders and substitute Felipe Carvalho forced the ball towards goal and Boyata could only deflect the ball agonizingly over the line.
Celtic had no response and their Champions League dream faded away.