Daniel Larsson grabbed the only goal of the game at Ibrox, leaving the Scottish champions with a major hurdle to overcome in next Wednesday's away leg in Sweden.
It was another frustrating encounter for Ally McCoist, with the new Rangers manager still to secure his first competitive victory at the Light Blues helm.
Lee Wallace was handed his Rangers debut from the start, while skipper David Weir made his first appearance of the season after sitting out Saturday's Scottish Premier League opener against Hearts through suspension.
Kirk Broadfoot and Juan Manuel Ortiz were the players who were sacrificed and both had to settle for a place on the bench.
Before the match kicked off, fans were warned about the repercussions of any sectarian singing, having already been banned for next week's return leg in Sweden after being found guilty of discriminatory chanting in European matches last season.
Rangers almost suffered the worst possible start to the tie when Larsson raced past Madjid Bougherra and tried his luck with a shot in the opening minute of the game that trundled across the gaping goal and wide of target.
Malmo were still looking dangerous and Allan McGregor was called into action to block a ferocious shot from the right flank from Wilton Figueiredo, before Jimmy Durmaz blasted the rebound past the post.
Rangers failed to heed the warning signs and the Swedes surged into the lead after 18 minutes.
The move began when Steven Whittaker was robbed of possession by Durmaz and the ball was eventually laid on by Figueiredo to Larsson, who rifled an unstoppable drive past the grasp of McGregor.
Rangers made their first change with just half an hour gone when Weir was replaced by Ortiz, with Sasa Papac moving into the centre-half position vacated by the captain and Wallace dropping back to left-back.
Figueiredo then picked up the first booking of the game for handball and Steven Davis saw his resultant free-kick palmed to safety by goalkeeper Dusan Melicharek as Rangers headed into the interval to the sound of jeers from the crowd.
The mood was lifted at half-time when new signing Dorin Goian - unable to feature in tonight's game as he awaits a work permit and international clearance - was paraded on the pitch to enthusiastic applause during the break.
Rangers emerged for the second half eager to haul themselves back into the tie but it was Malmo who almost had the net bulging again just a few minutes after the restart.
Papac was cautioned for tripping Larsson before Figueiredo's curling free-kick was tipped one-handed over the crossbar by a full-stretch McGregor. Agon Mehmeti then sent a header flashing narrowly wide of the upright from the corner.
Lee McCulloch was not too far away with a deflected shot from distance but McGregor was forced to come to Rangers' rescue again when he denied Figueiredo, who was allowed to run unchallenged before unleashing the shot. The goalkeeper was also alert to block a free header from Daniel Andersson.
Rangers should have been back on level terms when Nikica Jelavic whipped a tempting ball into the path of Steven Naismith, who should have buried the shot from close range, but he connected on his weaker foot and mis-kicked wide instead.
Naismith had the chance to make amends just moments later in what was a carbon copy of the previous move and, unfortunately for him, the finish was just as poor the second time around.
It was proving to be a busy night for McGregor - who, thankfully for Rangers, signed a new long-term contract this summer - and he did well to block from Jiloam Hamad in the final minutes of the match and keep his side's hopes of progression alive for now.