An own goal from Abdoulaye Meite gave Villa an early lead before Peter Odemwingie levelled for the Baggies with his 14th goal of the campaign.
Albion midfielder Paul Scharner was sent off after 62 minutes for two bookable offences, but Mulumbu netted the winner with six minutes remaining.
West Brom have now taken 16 points from nine matches under head coach Roy Hodgson.
Hodgson was forced to make three changes with skipper Chris Brunt and winger Jerome Thomas (both hamstring) and defender Steven Reid (calf) all ruled out.
Villa caretaker boss Gary McAllister recalled Gabriel Agbonlahor in place of Emile Heskey, who had a slight knee problem.
Albion fell behind via the first meaningful attack of the game after four minutes.
Agbonlahor held the ball up inside the Baggies box before picking out Stewart Downing.
Meite was unable to deal with his low cross and deflected the ball past Scott Carson into the corner of the net.
Carson was then unable to cling onto a deflected shot from Villa skipper Stiliyan Petrov, but he was able to recover in time and grab the rebound before Darren Bent could take advantage.
Scharner became the first player to be booked for a challenge on Nigel Reo-Coker.
The home side threatened for the first time when Brad Friedel turned aside a curling 25-yard drive from Mulumbu at the expense of a corner.
Ashley Young seized onto a lack of control by Gonzalo Jara and cut inside, but he curled his shot past the far post.
Carlos Vela then missed a great chance to bring Albion level, the on-loan Arsenal striker latching onto Simon Cox's defence-splitting pass but failing to keep his shot down as he tried to chip Friedel.
Downing created space for himself after cutting inside but dragged his left-footed shot past the post.
The England winger then teed up the overlapping Kyle Walker, whose menacing cross was tipped over the bar by Carson.
More good play by Downing created an opening for Agbonlahor whose shot was deflected wide.
Both sides were forced into a defensive switch after 50 minutes with Jara and Dunne hobbling out of the action to be replaced by Gabriel Tamas and Ciaran Clark respectively.
Albion fans appealed optimistically for a penalty but Luke Young clearly got a touch on the ball to deflect it away from Vela and back to Friedel.
But after an hour Odemwingie brought West Brom level.
Morrison's deep free-kick was headed back across goal by Jonas Olsson and Cox and Scharner miskicked before Odemwingie lashed the ball into the roof of the net.
Two minutes later Albion were reduced to 10 men when Scharner hauled down Petrov and was sent off.
Carson parried away a curling attempt from Ashley Young as Villa looked to make the most of their numerical advantage.
Morrison blocked a fierce attempt from Agbonlahor, who was then booked after fouling Mulumbu.
Collins deflected a fiercely struck cross from Walker wide and saved a low drive from the impressive Downing.
Mulumbu was booked for bringing down Bent but then put his side in front after 84 minutes, his sixth goal of the season.
He ran onto Cox's pass and evaded Luke Young before nicking the ball past Friedel under pressure from Clark.