Hart's position as England's No.1 remains secure and, barring unforeseen circumstances, he will be in goal against Moldova on Friday and again in Ukraine next Tuesday.
But he was criticised for his performance in last month's friendly victory over Scotland and also came under fire following City's Premier League defeat at Cardiff.
City manager Manuel Pellegrini even warned Hart risked losing his place unless he was "quickly in the best shape again".
But Milner, part of a City side which kept more Premier League clean sheets last season than any other team, said: "He's a victim of his own success.
"People said he had a poorer season last season but he won the Golden Glove.
"When we played Borussia Dortmund last year (in the Champions League) it was one of the best goalkeeping performances I've ever seen.
"We could have lost that game 8-1 very easily and that's not exaggerating.
"I am blessed as a midfielder because if I miscontrol the ball I've got four defenders and a goalkeeper to save me.
"If Joe makes a mistake it's going to end up in the back of the net. That's the unfortunate thing about being a goalkeeper.
"But he is a top keeper and we're lucky to have him."