Jonny Evans' first-half header brought the scores level in the tie and it looked like being enough until a dramatic finish in extra-time.
Phil Bardsley's goal in the final minute of extra-time after a shocking mistake by David De Gea looked to have sent Sunderland through.
However, Javier Hernandez sent the game to penalties with a second goal for United in the dying seconds.
Vito Mannone was the hero for Sunderland in the shoot-out as he saved Rafael's fifth penalty as Gus Poyet's side progressed 2-1 on penalties.
Danny Welbeck, Craig Gardner and Phil Jones all put their spot-kicks over the crossbar, while Mannone saved an effort from Adnan Januzaj, with Darren Fletcher the only scorer for United.
Marcos Alonso and Ki Sung-Yeung beat De Gea from the penalty spot - and they proved enough to book a Wembley date with Manchester City on March 2.
United started on the front foot and went close to scoring after just six minutes when Januzaj picked out Hernandez with a free-kick and the Mexican's downward header was well kept out by Mannone before his defenders cleared the ball to safety.
Sunderland created their first noteworthy chance of the game on 19 minutes when Johnson found Fabio Borini with a superb pass upfield and the on-loan Liverpool forward flashed his half-volley just over the crossbar.
The visitors were defending well and putting plenty of pressure on United in possession as they restricted them to not many goalscoring opportunities in the opening half hour.
United almost made the breakthrough on 36 minutes when Darren Fletcher got on the end of Shinji Kagawa's cross and was unlucky to see it come back off the post and the rebound fell to Welbeck, who saw his effort superbly saved by Mannone.
A minute later United did grab the crucial opener as Welbeck flicked his shot from a corner goalwards and Evans ghosted in unmarked losing his marker John O'Shea to head home from close-range.
Sunderland responded well to going behind and Wes Brown went close when he rose highest to meet Johnson's corner, but Alexander Butter was on hand to head away the ball inside the six-yard box.
In the second half Januzaj looked the man most likely for United and he wasted a good opening on 69 minutes when substitute Antonio Valencia picked him out on the edge of the box, but the midfielder sliced his shot wide of the target.
Sunderland started to come more into the contest in the closing stages and Johnson went close 14 minutes from time when his volley deflected wide of the target and Alonso fizzed a half volley just wide of the target.
Neither side could find a route to goal and the game was forced into extra-time.
In the first period of extra-time Hernandez wasted a great chance to double United's lead when Januzaj sent the Mexican clear on goal and with only Mannone to beat the striker curled his shot horribly wide of target.
The game looked to be petering out in the second half of extra-time until a minute from the end of the extra 30 minutes when Bardsley tried his luck from the edge of the box and De Gea somehow let it squirm from his grasp and into the net to send the away fans into raptures.
However, there was still time for a dramatic finish as Januzaj swapped passes with Smalling inside the box and he crossed for Hernandez to fire the ball home into the net and save De Gea's blushes and force the game into a penalty shoot-out.
In the shoot-out United missed four of their spot-kicks with Rafael seeing the fifth penalty saved by Mannone as Sunderland recorded a famous 2-1 success on penalties to set up a final with free-scoring City.