Blues goalkeeper Maik Taylor made the decisive save to deny Bees defender Craig Woodman from the spot.
The Bees looked to have followed up their third-round victory over Everton with another win after Sam Wood's 15-yard drive midway through the second half.
But striker Kevin Phillips broke the League One side's hearts in the 92nd minute with the equalising goal.
Brentford against defied their poor league form - they are currently 19th in the table - with their performance.
Birmingham manager Alex McLeish made seven changes from the side which had defeated Blackpool at the weekend.
Blues midfielder Michel had the first attempt on goal but his low 25-yard drive was a couple of feet off target.
Craig Gardner's fine pass found Phillips in space but the ball appeared to get caught under the foot of the veteran and the chance went begging.
Then City captain Stephen Carr's persistence won him the ball in the Brentford box and it dropped invitingly into the path of Phillips. But his low, first-time shot lacked power and did not extend Bees skipper Richard Lee.
Lee kept his side on level terms with a fine reflex save after 25 minutes.
He blocked a powerful drive from Stuart Parnaby at close range after he had got on the end of a cross from O'Connor.
But Brentford retaliated almost immediately and Taylor had to be alert to turn Myles Weston's left-footed effort around the post after he had outpaced Scott Dann.
Blues ended the half with a flourish and Gardner created an opening for Matt Derbyshire - but the striker elected to try to find O'Connor rather than shooting himself.
Lee got his body behind a 30-yard drive from Gardner early in the second half after good play by Phillips.
Brentford sensed the chance of an upset and Ridgewell volleyed a menacing cross over his own bar from Charlie MacDonald.
Michel came to City's rescue when he cleared a volley from Michael Spillane off the line from a Wood corner.
McLeish made his first change after 63 minutes with O'Connor replaced by Nathan Redmond - and the youngster made an instant impact with his first-time pass picking out the run of Lee Bowyer, who dragged his shot just past the post.
But after 69 minutes Sam Wood gave Brentford a shock lead. A mistake by Carr allowed Weston the opportunity to cross and the ball was headed back by MacDonald to Wood, who made no mistake with his 15-yard drive.
McLeish brought on Nikola Zigic for Derbyshire in an attempt to rescue the tie.
Lee kept the Bees ahead by making a finger-tip save to turn aside Gardner's curling effort from the edge of the box.
But the woodwork rescued Blues when Woodman's drive hit the top of the bar.
Zigic could have levelled but he headed over from Carr's cross.
Then in injury-time Phillips rescued Birmingham from close range after being teed up by Ridgewell.
Birmingham tried to take advantage of their reprieve in extra-time and Lee made a fine save to keep out Zigic's header from Bowyer's centre.
Phillips was then only inches wide with a 20-yard drive, before Gardner was just wide with a low free-kick in the last minute.
The game went to penalties and Phillips, Gardner, Scott Dann and Carr scored from the spot for the home side, with Bowyer shooting over the bar.
Kevin O'Connor, David Hunt and Lewis Grabban netted for the Bees but Spillane hit the bar, before Taylor denied Woodman.