Jarvis' fifth goal of the season earned Wolves a first win over their derby rivals in 31 years and took them to just a point behind Villa.
Villa fans chanted 'We want Houllier out' at the final whistle and greeted his walk to the tunnel with loud boos.
Houllier had incurred the wrath of some Villa fans recently when he fielded a much-changed side for the FA Cup clash at Manchester City.
Then this week the former Liverpool boss has had to deal with the fall-out from the Richard Dunne-James Collins bust-up with coaching staff at a team bonding session.
But this defeat will have hurt the Frenchman as it has sucked Villa right into the heart of a survival battle.
They have taken only five points from the last six games.
It was only Wolves' second away victory of the campaign but Mick McCarthy's side are now unbeaten in four games.
Before kick-off a banner was unfurled on the Holte End with the message 'Had enough, Houllier out' but it was quickly taken down by a steward.
Wolves made a positive start and went close to taking the lead after two minutes.
Winger Adam Hammill got past Villa left-back Nathan Baker and squared the ball back to Nenad Milijas whose low 20-yard shot was turned aside by Brad Friedel.
The visitors were looking full of confidence and Kevin Doyle had a low drive blocked by Kyle Walker.
Friedel then went down at the foot of his post to keep out a Hammill effort, with Jarvis in a good position in the middle.
Play was flowing from end to end and fit-again Villa defender Carlos Cuellar headed over from an Ashley Young corner.
Wolves defender Christophe Berra headed in a Hammill centre after 21 minutes, but the effort was ruled out for offside by referee Phil Dowd.
Doyle also had the ball in the net after half an hour with a far-post header, but the cross from Hammill had drifted out of play.
Wolves defender Richard Stearman was yellow-carded after 31 minutes for a foul on Young.
Houllier made a change after 33 minutes with Baker, who was showing no visible signs of carrying an injury, replaced by Fabian Delph at left-back.
Berra left the pitch, presumably for some treatment, but returned to play a part in Wolves taking the lead after 38 minutes.
Milijas delivered the cross from the right, Berra flicked the ball on to Jarvis whose first-time shot from 15 yards flew past Friedel into the corner of the net via a post.
It was Jarvis' fifth goal of the campaign and fourth in the Premier League.
Jean Makoun should have put Villa level on the stroke of half-time.
The former Lyon player was completely unmarked but headed Stewart Downing's centre wide.
Villa started the second half with positive intent and were awarded a free-kick in a dangerous position right on the edge of the Wolves box after Stearman brought down Bent.
Television replays suggested the offence had taken place inside the penalty area.
Bent's eventual shot was deflected for a corner after Young had rolled the free-kick into his path.
Stearman made a vital block on a fiercely-struck Downing volley before Houllier made a double change after 61 minutes.
He brought on Robert Pires and Gabriel Agbonlahor in place of Reo-Coker and Albrighton.
But the substitution of Albrighton was greeted with loud boos from Villa fans and cries of 'You don't know what you're doing' towards Houllier.
McCarthy replaced Hammill with Sylvan Ebanks-Blake three minutes later.
Friedel beat out a dipping 25-yard drive from Milijas after Villa were guilty of giving the ball away in midfield.
Young was trying his best to bring Villa back into the game and the woodwork came to Wolves' rescue after 80 minutes when his shot crashed against the crossbar.
George Elokobi was then booked for chopping down Downing.
Cuellar volleyed over after Pires had knocked the ball down into his path, but Villa could not find an equaliser.