Jordi Gomez, Adam Johnson and Connor Wickham struck in the final 13 minutes to stun the hosts and give the Premier League side victory at St Andrew’s.
But Blues were angered after Gomez’s opener, claiming the ball had gone out in the build-up.
The Black Cats made hard work of the win as David Cotterill hit the post and Clayton Donaldson missed for the hosts.
It was harsh on Birmingham, who had matched the Barclays Premier League side in Lee Clark’s first meeting with Sunderland as a manager, after a two-year spell with the Black Cats which ended in controversy in 1999.
The former midfielder was forced to leave after wearing an offensive T-shirt which mocked Sunderland fans.
Clark called on Colin Doyle in goal while opposite number Gus Poyet made six changes from the side which held Manchester United to a 1-1 draw at the weekend.
Costel Pantilimon, Billy Jones and Gomez made their first starts for the Black Cats.
Sebastian Larsson also started, having spent five years at Blues before moving to the north-east.
Paul Caddis tested Pantilimon from distance early but any early hope for a Birmingham onslaught was swiftly dashed.
Blues’ grit and determination gave them a foothold but they rarely breached the Sunderland backline with serious menace in the opening half an hour.
The game’s pedestrian nature did little to conjure a spectacle either.
The visitors, disjointed and ragged, rarely looked like a top flight side with Jozy Altidore isolated up front.
Doyle was well protected by the Birmingham defence and limited and limp Sunderland lacked the nous to penetrate.
The hosts created the best chances late in the first half though and could have gone into the break ahead.
First Wes Thomas turned Mark Duffy’s cross wide on 43 minutes and 60 seconds later Cotterill hit the outside of the post from a tight angle.
They were brief bright moments in a dour first half and things did not improve in the early stages of the second half.
While Birmingham remained the aggressors, their final ball rendered any build-up useless with Sunderland easily holding firm.
Gradually the top flight side began to take charge through.
Gomez could not force the ball in after a 65th minute scramble and Doyle saved Liam Bridcutt’s low effort a minute later.
However, the Black Cats broke the deadlock with 13 minutes remaining – when Gomez received the ball 25 yards out and rifled into the bottom corner.
Birmingham were unhappy though as they felt the ball had gone out deep in the Sunderland half during the build-up.
Donaldson wasted a fine chance to instantly level when he shot at Pantilimon and David Edgar shot over before Sunderland sealed the game.
Johnson fired in after Altidore’s shot was saved in the 87th minute, before Wickham stroked in from 15 yards a minute later.