Phil Bardsley set the Black Cats on the road to victory, just their fourth of the season and third in the same competition, after finding the net for the first time since a 1-1 draw with the Saints in May.
The former Manchester United full-back struck from close range 14 minutes into the second half after former Sunderland goalkeeper Kelvin Davis had parried Jozy Altidore's header.
Substitute Sebastian Larsson looked to have wrapped up the win with a second four minutes from time, but defender Maya Yoshida's late header set up a tense conclusion.
In a game which started slowly, the Black Cats who arguably enjoyed the better of the early chances, although neither Davis nor opposite number Vito Mannone were unduly busy in front of a muted crowd of just 15,966 at the Stadium of Light.
The Saints' keeper had to block Craig Gardner's 48th-minute snapshot with his legs, but was otherwise largely untroubled apart from the goals.
However, Mannone enjoyed a let-off 16 minutes from time when he blasted a clearance against substitute Adam Lallana and saw the ball drop just wide of his unprotected net.
The Black Cats became the first team to defeat the Saints in nine attempts in all competitions since August with Bardsley's effort just the third goal they had conceded in the process.
Just quite what the selections of the respective managers said about their priorities for the season was a topic for debate with Mauricio Pochettino replacing 10 of the men who started at Stoke at the weekend and Poyet retaining only seven of his strongest side, although that was in part dictated by injury and suspension.
The Saints, however, started with a fluency which left their hosts largely chasing shadows in the early stages with Gaston Ramirez and Jay Rodriguez linking well with lone striker Tadanari Lee and the impressive Jack Cork conducting proceedings from a deep-sitting midfield berth.
Lee was unable to make the most of full-back Calum Chambers' fourth-minute cross under pressure from John O'Shea and then fired over from distance two minutes later after a corner had been recycled.
Altidore, whose only goal to date for the club came in the second round of this competition, fired tamely at former Black Cats goalkeeper Davis with 10 minutes played, while Emanuele Giaccherini miskicked in front of goal from Ondrej Celustka's cross five minutes later.
However, much of what followed was uninspiring - and it was not until the final minute of the half that the deadlock was almost broken.
Cork took his time to pick out Ramirez in space beyond the far post, and he steered a left-foot shot towards Vito Mannone's top corner, but crucially just high and wide.
The Black Cats got off to a much livelier start after the break and might have gone ahead within three minutes when Altidore's clever flick inside fell perfectly for Gardner, and Davis had to block the midfielder's first-time shot with his legs.
But the opening goal eventually arrived with 59 minutes gone when Johnson floated a free-kick to the far post where Wes Brown, making his first competitive start in 21 months, rose unopposed to turn the ball back across goal for Altidore to power in a header.
Davis could only parry and Bardsley stabbed the rebound home from close range despite the best efforts of Jos Hooiveld on the line.
All Sunderland's good work might have been undone with 74 minutes gone when Mannone drilled a clearance straight into Lallana and with no prospect of recovering his ground, saw the ball spin just wide of the goal.
But Larsson's 86th-minute strike from Altidore's intelligent pass ensured there was no way back for the visitors despite Yoshida's close-range header two minutes later.