Hugh Robertson MP believes the entire bidding process for both the 2018 and 2022 World Cups was "strange and obscure".
He said the turning point against England's bid came when Blatter addressed delegates just before the vote and reminded them of several less-than-complimentary stories about FIFA that had been pursued by the British media.
FIFA are holding a meeting this week to discuss moving the 2022 tournament in Qatar to winter because of the stifling summer heat.
Robertson told Sky Sports News: "I didn't agree with the way the competition was run but there was a competition: Qatar won it, and congratulations to them.
"The ambition to take football to parts of the world that haven't staged major tournaments before is the right one.
"If we're going to do that, we're going to have to make some compromises over the timings of the English leagues, the North European leagues.
"The interesting thing is that there was nobody there during that bidding process who really thought that staging that World Cup in the summer was a sensible option, so to me it was always obvious it was going to be moved to the winter."
Robertson believes that the bidding process as a whole for 2018 - won by Russia - and 2022 was flawed.
"It was a strange and obscure bidding process; had it all been made clear I suspect we might not have bid," he said.
"There were a lot of FIFA delegates who looked at either the Prime Minister or Prince William in the eye and promised their vote, and then didn't deliver.
"Thirdly, there was a slightly unsatisfactory chapter. I've been told by a number of people in the room that Sepp Blatter, straight before the vote, stood up and reminded the delegates what the British press had done to them in the period running up to the vote.
"That must have influenced the way some of those delegates who promised to give us their vote then didn't. Those three things make me feel very uneasy as a Sports Minister about the whole process."
The chairman of the Football League, meanwhile, believes FIFA should run the vote for the 2022 World Cup again rather than switch it to the winter.
Greg Clarke said: "My view is that it should be like any public tender process and if the tender isn't valid, then have a new one.
"Don't start fiddling at the edges. If the tender doesn't work, run a new one.
"There were some really good bids from people like Australia and the USA who spent a lot of money and have a great footballing culture and really wanted to run the World Cup, and they were excluded on the grounds that it was going to be in the summer and that it was going to be in Qatar.
"If I was them I would be less than happy if FIFA are now changing the rules, and my view is: run it again."