Four-time champions Italy, two-time winners Uruguay and dark horses Costa Rica join England in Group D in Brazil.
Roy Hodgson's side have to travel thousands of miles and play in the sweltering heat of the jungle as they attempt to make it through to the knockout stages.
"It's not easy, but it's a World Cup and everybody has earned the right to be there," Milner said.
"They are all good teams, and if you go there with a so-called easier group, you might not come out of the box as fast as you need to.
"We know we have to be on top of our game in the first game of the group, and that could be an advantage, hitting top gear straight away, and then carry through the tournament like that. It could work in our favour."
But before Milner heads to South America, he wants to help City lift the Premier League trophy.
Manuel Pellegrini's side have a 100 per cent home record but have struggled away from home with just two wins to show from eight matches on the road.
The latest frustration came in Southampton on Saturday, where City were held to a 1-1 draw after a poor performance.
"Looking at our overall form away from home, I don't think it has been too different," Milner said.
"The results have been different, but in the majority of games we have played away, the way we have dominated possession and deserved to win the game is similar.
"But we have somehow ended up losing games through mistakes and missed chances. Saturday was one of the exceptions, because we didn't play well and got a point out of it.
"We took the lead and that is a big thing, but it's disappointing as we started the better, but after the goal we took our foot off the gas, and didn't stay on top of them."