The Millers added the scalp of Leicester to those of Sheffield Wednesday, Wolves, Southampton and Leeds in cup competitions this season.
But they waited until first-half injury time to take the lead at the Don Valley Stadium with a goal of classic simplicity, adding a second just after the break.
Neither team shone in a disappointing first 45 minutes with Leicester's best chance falling to Barry Hayles after 41 minutes. He went through on a solo run, but he delayed his shot and allowed Dale Tonge to get back with a covering challenge and he put his shot wide of the target.
And the miss proved really costly as the Millers swept into the lead with a peach of a goal when Pablo Mills was given all the time in the world on the right side of mid-field to pick out Drewe Broughton on the far post with a perfect pass and Broughton side-footed the ball past Paul Henderson in the City goal.
Leicester made two changes after the break, but in the 58th Rotherham doubled their lead and again Mills was the architect. He threaded a pass into the City penalty area where he picked out Tonge who lifted the ball over the advancing keeper and into the net.
And when Andy Warrington saved Matty Fryatt's injury-time penalty, City knew it simply wasn't their night.