The Magpies were ahead in the 18th minute when poor Villa defending allowed Hatem Ben Arfa the chance to convert.
The hosts levelled through Christian Benteke's header in the 67th minute, but Newcastle went back in front in the 73rd when Brad Guzan palmed away the impressive Ben Arfa's shot and substitute Gouffran pounced.
There has been plenty of talk about doom and gloom around Newcastle after a summer that only yielded one signing in loan capture Loic Remy, who made his first start for the club here.
But since they lost their first Premier League game of the campaign 4-0 against Manchester City, the three other top-flight fixtures for Alan Pardew's men this term have seen them draw 0-0 with West Ham, beat Fulham 1-0 and then record this victory over Villa.
Villa, on the other hand, began the season with a 3-1 win at Arsenal, but have now lost three league matches in a row.
A good tackle from Mathieu Debuchy took the ball away from Gabriel Agbonlahor in the box early on, and while each side did their best to probe in the opening exchanges, neither had much joy in finding a way through.
That changed just after the quarter-hour mark when slack Villa play allowed Newcastle to snatch the lead.
Karim El Ahmadi was hustled into losing possession and the ball was picked up on the left by Remy, who managed to evade Matthew Lowton and fashion a cut-back into the danger zone.
There were several Villa players in the box but the pass came via Papiss Cisse to Ben Arfa, and the France midfielder had little trouble finishing from close range.
Villa responded in a positive manner, with Mapou Yanga-Mbiwa having to do well to halt Benteke's progress and Andreas Weimann seeing his shot in the area blocked by Fabricio Coloccini.
The hosts then suffered another setback, with Jores Okore, who had injured himself in a challenge with Remy, having to come off, Ciaran Clark coming on in his place.
Fabian Delph was then shown a yellow card for scything down Ben Arfa.
Yohan Cabaye - back in the Newcastle starting XI after his proposed move to Arsenal did not materialise - and Remy tried their luck on the Villa goal with ambitious efforts that went well off-target.
Then, with four minutes of normal time remaining before the interval, Villa passed up a good-looking opportunity to level as Tim Krul failed to clear a cross with an attempted punch and the stray ball was struck into the side-netting by Clark.
Weimann dragged a shot wide soon after, before Cabaye cracked one across the face of the Villa goal just prior to the break.
Villa started the second half brightly and Antonio Luna made a bizarre decision in a useful position, electing to cross and seeing the ball cleared away when he might have shot.
Agbonlahor was then even more wasteful, somehow shooting wide right in front of goal from Weimann's cross when it seemed easier to score.
Newcastle looked to step things up and Guzan turned a Ben Arfa strike behind for a corner, from which Debuchy then fired wide.
Villa then sent on transfer deadline day signing Libor Kozak for his debut and, within seconds, they equalised.
A corner was swung in by Ashley Westwood from the right and Krul tried to claim it, but Benteke rose highest of everyone in the box to crash home a header, his fifth goal of the season in all competitions.
Villa were on level terms for just six minutes as Ben Arfa proved a thorn in their side once again, dancing forward and drilling a shot that Guzan could only parry.
Gouffran, who had come on for Remy, was on hand and had the space to bury the loose ball.