McSheffrey outjumped the Blades defence at the far post to head home Richard Keogh's flick-on to clinch Aidy Boothroyd's side their first back-to-backs wins of the season.
The Blades, who twice hit the woodwork in the first half through Stephen Quinn and Jamie Ward, were made to pay for failing to convert their chances in front of a crowd of 20,059, who witnessed their side's fourth home league defeat of the season.
Gary Speed's side, whose last home win was against Portsmouth in mid-September, are four points above the relegation zone.
Andy Reid, signed on loan from Sunderland yesterday, started on the bench for the Blades, who dominated the early stages.
Richard Cresswell fired off target in the third minute, Mark Yeates saw his shot from the edge of the penalty area blocked by Coventry defender Martin Cranie and Stephen Quinn hit the crossbar with a stunning 25-yard volley.
But Coventry struck the home side a blow with their first attack on the Blades' goal. Richard Keogh headed the ball back across the six-yard box and McSheffrey outjumped the home defence at the far post to head the ball into the top corner.
Quinn tried his luck with another volley from similar distance but failed to hit the target and after Sammy Clingan fired in a rare shot for the visitors from Lukas Jutkiewicz's cross, the Blades hit the woodwork again, this time when Jamie Ward's shot from 12 yards struck the post.
United were nearly made to pay again for their wayward finishing two minutes after the restart when Clingan's 30-yard free-kick forced Steve Simonsen into a fine save and Blades defender Nyron Nosworthy blocked Jutkiewicz's volley from the rebound.
The Sky Blues were full of confidence at this stage and Blades boss Speed sent on Reid for Quinn in a bid to unlock the visitors' well organised defence in the 57th minute.
The home side responded, with Cresswell going close with a header from Yeates' cross and Speed then sent on striker Ched Evans for defender Johnny Ertl.
But it was the visitors who threatened to extend their lead in the 70th minute when Clive Platt headed Clingan's free-kick the wrong side of the post, while substitute Marlon King, on for Jutkiewicz, curled a 20-yard free-kick wide in the 78th minute.
The Blades were full of endeavour in their bid to rescue a point in the closing stages, but apart from long-range efforts from Evans and Reid in the six minutes of time added on failed to create further clear-cut chances.