The Millers were undone by former striker Tom Pope who scored against his old club for the fifth time this season. After Louis Dodds opened the scoring for Vale after 55 minutes and Pope added the second eight minutes later.
Kieran Agard gave Rotherham hope eight minutes from the end, but Rotherham could not add to that as Vale held out against tremendous home pressure.
"We were obviously disappointed with the scoreline," Raynor said. "Vale came with a game plan to hit us on the break and they had two chances and they punished us. We had possession but we did not create the opportunities to match that.
"They were very physical and defended in numbers. We tried different formations and different substitutions but it didn't work on the day. We didn't seem to have that spark which is disappointing because it has been there in previous weeks.
"Conceding two goals as quickly as we did in the second half rocked us a bit, but if we could have got one back a bit quicker, who knows what could have happened.
"Tom Pope is punishing people this season and that is five goals he has scored against us. We have been inconsistent this season and consistency is the key to any season.
"We weren't at our best, but there is still a long way to go."
