Any hopes the Millers, who were Area Finalists last year, had of going one better this season were ruined early on as the League One side went two ahead inside seven minutes, and they were unable to turn things around despite a brave fight.
The first goal came after less than two minutes when an unmarked Peter Clarke headed in a left-wing corner on the near post, before Robbie Simpson nodded home his first Town goal five minutes later after Michael Collins' cross.
Lee Clark's men missed a number of other chances before Rotherham reduced the arrears after 35 minutes when Nicky Laws' trickery put him clear and his cut-back was converted by Adam Le Fondre for a fourth goal in as many games.
Rotherham could then have been level three minutes later when Law's free-kick was finger tipped round the post by Alex Smithies.
Andy Warrington produced some fine saves of his own to keep United in the hunt after the break and Kevin Ellison could have forced penalties had he not dragged an effort wide.