In a tightly-fought contest at the Ricoh Arena, it took a Harry Maguire own goal in the 57th minute to break the deadlock, but United drew level in the 80th minute through David McAllister.
An even first half saw Richard Cresswell shoot over the crossbar for the visitors in the first minute, while Coventry goalkeeper Joe Murphy did well to repel Michael Doyle's long-range volley.
The home side broke the deadlock just before the hour mark when midfielder Steve Jennings ran beyond the Sheffield United defence and was found by James Bailey.
Jennings' cross was aimed at David McGoldrick but was turned into the net by centre-back Maguire.
United drew level with 10 minutes to go when Doyle's shot was parried by Murphy and McAllister reacted first.
Maguire's night got even worse when it came to penalties as he saw his spot-kick saved by Murphy and then Ryan Flynn blasted over, allowing John Fleck to send Coventry through.
