The Millers were looking good value for a point until a quality finish from Republic of Ireland midfielder Jeff Hendrick floored them eight minutes from the end, although they twice went close to equalising in stoppage time.
Derby went into the game with nine of the team that started the opening game of last season and one of those men, Johnny Russell, made a surging run in the first 30 seconds that took him into the Rotherham penalty area before the ball was scrabbled clear.
Rotherham were struggling to cope with Derby's passing and movement and they were opened up again in the 16th minute when Craig Forsyth made a powerful run down the left but his cross just evaded Chris Martin.
But Derby were given a warning a minute later when a free kick from the left dropped invitingly to Kieran Agard 12 yards out and his shot on the turn was blocked by one of the home side's new signings Cyrus Christie.
Former Derby midfielder Paul Green shot wide from the edge of the box in the 21st minute but Simon Dawkins had a 20-yard drive deflected just past a post three minutes later.
It needed a brilliant reaction save by Adam Collin in the 26th minute to keep out Russell's close-range header.
Rotherham went close five minutes later when a Ben Pringle corner was met by a diving header from Karl Arnason, but the ball was cleared off the line by Will Hughes.
The visitors had come back into the game but Derby started the second half strongly with Christie fizzing the ball across the six-yard box with no white shirt able to apply a touch.
The home fans got their first sight of Real Madrid's on-loan midfielder Omar Mascarell when he replaced Hughes in the 53rd minute.
Rotherham made their first change when Jordan Bowery replaced Alex Revell in the 57th minute but it was Matt Derbyshire who had the chance to open the scoring when Pringle crossed from the left, but he headed wide from eight yards.
Some of the fluency had gone from Derby's game but an excellent pass from Mascarell released Christie down the right and his cross was bundled away before Martin could pounce.
But Derby finally broke Rotherham's resistance in the 82nd minute when they broke up the right and Christie played the ball inside to Hendrick, who swept a first time shot into the top-left corner from just inside the area.
Rotherham felt Pringle had been fouled by Christie but they responded impressively and Green had a header saved by Lee Grant in stoppage time before Agard's shot was cleared off the line by Forsyth.