Blues players and staff were incensed when Byron Webster scored to take the match into extra-time, when they felt the home side should have returned the ball to them after goalkeeper Colin Doyle had kicked the ball out of play following an injury to Birmingham defender Dan Burn.
Yeovil moved ahead in extra-time through Luke Ayling, but then immediately allowed the visitors to walk in an equaliser from Lee Novak, apparently as compensation for Webster's controversial effort, before Tom Adeyemi's penalty in the shoot-out took the 2011 League Cup winners into the third round.
In an even first half, it was Blues who started the brighter of the sides and snatched a deserved lead on 20 minutes when Kyle Bartley - on loan at St Andrew's from Swansea - rose above his marker and headed home Neal Eardley's corner.
However, the visiting fans hardly had time to celebrate the opener as Ed Upson rifled a stunning 20-yard free-kick past Doyle two minutes later to draw level.
As the half progressed Blues again threatened, with Wayne Hennessey pulling off a fantastic save from Adeyemi's close-range effort before Andrew Shinnie fired over.
But Shinnie - who signed from Inverness in the summer - scored a memorable first goal a minute before the break after he lashed home Chris Burke's fizzing cross.
Birmingham should have wrapped up the tie 10 minutes later when Matt Green flashed Eardley's cross narrowly wide, before he again eluded the Yeovil defence but shot straight at Hennessey.
But huge controversy unfolded in the last minute of normal time when Blues keeper Doyle kicked the ball out of play as Burn received treatment.
With Burn still down and referee Darren Sheldrake allowing play to go on, Webster quickly received the ball from the throw-in and fired the ball into the empty net.
Birmingham were incensed at Webster's decision not to give the ball back - sparking ugly scenes between both sets of players before the game entered extra-time.
The Glovers' longest-serving player Ayling raised the roof of Huish Park on 104 minutes after lashing a sensational left-footed drive into the top corner to notch his first goal for the club.
But straight from the restart, the Yeovil home crowd was left stunned when their side bowed to the laws of sportsmanship by allowing Novak to run the ball into the net completely unchallenged to equalise.
It was the second time Glovers chief Gary Johnson has instructed his side to concede in this manner - with a similar scenario taking place in August 2004 during Yeovil's 3-2 Carling Cup win over Plymouth.
Novak should have won the game in normal time for Lee Clark's men when he dinked the ball over Hennessey - but the ball rolled agonisingly wide.
However, Birmingham held their nerve during the shoot-out, with Adeyemi's final penalty coming down off the crossbar and creeping over the line.