Carsley, who was working as development coach with the Sky Blues, links up with new Blades boss and former Everton team-mate David Weir.
The 39-year-old leaves the midlands club after two playing spells, retiring in 2011 to move into coaching.
He quickly progressed through the coaching ranks from the Under-18s to City's Under-21 side, before becoming involved in the first-team set-up and even taking the reins as caretaker boss last season after the departure of Mark Robins.
Current Coventry manager Steven Pressley said: "Lee has been a highly valued and tremendously popular figure among both the club's players and staff and I have certainly enjoyed working with him since I arrived here.
"I do though fully appreciate that this opportunity is one he could not turn down and I wish him nothing but the best of luck and offer my heartfelt congratulations on his new role."
Carsley's official title at Bramall Lane is assistant manager (technical) and he will work alongside assistant manager (performance) Adam Owen.
Weir told United's official website: "Lee is someone I trust immensely and I'm delighted that we've been able to recruit him. His experience at this level will prove invaluable to us and he was successful as boss of Coventry in a short spell last season."