After making all the early running, the Millers suddenly went off the boil and allowed Luton to grab a shock lead after 20 minutes.
Tom Craddock was the scorer from the penalty spot, the striker making no mistake from 12 yards after Andy Nicholas had brought down Claude Gnapka.
Rotherham huffed and puffed and finally drew level in first half injury-time. Ryan Taylor picked up a loose ball, sent Tom Pope away on the right and his cross was turned in by Adam Le Fondre from close range.
After 62 minutes, Rotherham looked to be on their way to round three when Stephen Brogan headed them in front with a looping header from a Gary Roberts cross.
But the home side were caught napping again 11 minutes later when they lost possession deep in their own half and Kevin Gallen fed substitute Mark Nwokeji who drilled a low shot past Andy Warrington.