The Millers led at half-time through Joe Skarz's goal but the Sky Blues replied after the break with efforts from captain Carl Baker either side of Franck Moussa's strike.
The visitors almost made the perfect start after three minutes when Blair Adams capped a fine solo run with a shot that hit the post, but after that the first half belonged to Rotherham.
They had to wait until the 35th minute for a goal though, but forced four good saves out of Joe Murphy and saw three other good opportunities go begging.
But the breakthrough finally came following a short free-kick by James Tavernier. With Coventry expecting a ball into the area, he rolled the free-kick out wide to Skarz and his shot took a deflection off a defender to take it wide of Murphy and into the back of the net.
So vociferous were Coventry manager Steve Pressley's protests following the goal that he was ordered to the stands by referee Gary Sutton.
The visitors stepped up the pace after the break, forcing four corners in the first six minutes and they got their reward in the 71st minute. Rotherham skipper Craig Morgan received a straight red card for his tackle on Callum Wilson and Baker equalised from the resulting spot-kick.
And the Sky Blues made Rotherham pay dearly five minutes from time when Baker's corner was headed goalwards by Moussa and deflected into the net off a defender.
The Millers threw away a chance to redeem a point when they were awarded a penalty in the 11th minute of added time, but substitute Michael O'Connor's spot-kick was saved by Murphy.
And with the Rotherham keeper having gone forward as the ball broke free, Baker scored his second of the game - stroking the ball into the empty net from the half way line.