Steven Whittaker and Madjid Bougherra were both shown straight red cards, before Malmo's Ricardinho was also dismissed by Russian referee Vladislav Bezborodov on a night of drama at the Swedbank stadium.
Nikica Jelavic cancelled out Daniel Larsson's strike from last week's clash in Glasgow but it was Jiloan Hamad who had the final say, with Rangers forced to settle for a Europa League play-off spot instead of taking a step closer to joining Europe's elite in the Champions League group stages.
Rangers were dealt a blow ahead of the match with the news that Dorin Goian was ruled out through injury just hours after being cleared by UEFA to take part in the clash.
The Romanian missed the first leg at Ibrox because his work permit was not granted in time and he was initially ruled out of the second leg too before Rangers eventually succeeded with an appeal.
However, the club's official website said Goian had a hamstring problem, meaning Sasa Papac was drafted in to cover at centre-half, with skipper David Weir also injured.
Juan Manuel Ortiz was added to the starting line-up, with manager Ally McCoist making one change from the side who defeated St Johnstone in the SPL at the weekend.
Rangers started strongly and could have opened the scoring when Ortiz slipped into the path of Jelavic in front of goal, but Daniel Andersson was alert and hooked to safety before the striker could unleash the shot.
Jelavic was threatening again when he rose above everyone else in the box to meet a Steven Davis corner, but his header was prevented from finding the top corner by goalkeeper Dusan Melicharek, who punched the effort wide.
At the other end, Pontus Jansson should have done better when the ball broke to him just inside the box but he blasted well over the crossbar, before goalkeeper Allan McGregor was called into action to tip a swerving Wilton Figueiredo free-kick over.
Rangers' task of hauling themselves back into the tie was made even more difficult when Whittaker was shown a straight red card after 18 minutes for throwing the ball at Figueiredo after the referee had given a free-kick for a Bougherra foul on Larsson.
Despite the numerical disadvantage, it was Rangers who took the lead with 23 minutes gone. Maurice Edu provided the cross and Jelavic finished well with a volley that fell beyond the grasp of the goalkeeper.
Malmo could have been back on level terms on the night but McGregor - who had produced several impressive saves in the first leg - came to the rescue with a great block to deny Jimmy Durmaz, before Lee McCulloch cleared the danger.
There was drama on the stroke of half-time when Durmaz, who was fouled by McCulloch, then appeared to kick out at the Ranger player but he escaped unpunished, while McCulloch was booked for the initial challenge.
Larsson was the main threat at Ibrox and he had the chance to add his name to the scoresheet again when he met Figueiredo's lay-off but his shot was weak and easily dealt with by McGregor.
Rangers then had another player sent off after 65 minutes when Bougherra was shown a straight red card for a challenge on substitute Dardan Rexhepi.
The referee indicated that the Rangers defender had elbowed his opponent, who left the pitch on a stretcher.
The crime count continued to rise when Ricardinho picked up a second yellow card after 72 minutes for a foul on Jelavic and Malmo also saw their numbers reduced.
However, it was the Swedes who almost grabbed what could have been the decisive goal when Figueiredo unleashed a ferocious shot that fizzed narrowly over the crossbar.
McGregor pulled off yet another superb save to deny Jansson but the Rangers goalkeeper was unable to prevent a curving effort from Hamad from finding the back of the net with 10 minutes to go to secure Malmo's passage to the play-offs.