The Millers, who have beaten Sheffield Wednesday, Wolves and Southampton in the Carling Cup this season, took the lead after 17 minutes when a long throw by Andy Nicholas picked out central defender Ian Sharps and his back header looped over Leeds goalkeeper David Lucas and into the back of the net.
Leeds equalised when Jonathon Howson slotted home from close range, but three minutes after half-time Reuben Reid was fouled by Andy Hughes in the penalty area and although Mark Hudson's spot-kick was saved by Lucas, he followed up to slam home the rebound.
Then two goals in five minutes at the start of the second half stunned Leeds.
After 48 minutes Drewe Broughton headed home an inviting cross from Reid and five minutes later Nick Fenton headed home Alex Rhodes' corner.
Although Leeds pulled one back three minutes later when Enoch Showunmi beat Andy Warrington with a powerful drive, Rotherham withstood a late Leeds onslaught for a memorable victory.