Downing lines up against Aston Villa on Saturday in an Upton Park encounter between two sides who are struggling for Premier League consistency.
The winger has been criticised in some quarters for underachieving after winning 34 caps for his country - but Allardyce said of his £5m man: "Other people's opinion is of no interest to me. My opinion is the only opinion that matters between me and Stewart Downing.
"Stewart Downing had a difficult pre-season where he wasn't sure where he was coming or going. Finally he made a decision on West Ham United which we were delighted with and there's very good progression and much more contribution as the more games he plays.
"The more and more quality we see on a much more regular basis. There's a settling-in period like anybody else. People divide their opinion far too quickly in this game. Real Madrid have done it with Gareth Bale, haven't they?
"He's no good and he's a waste of money and all that is coming out of there now and then he scores two and now he's a hero.
"We all know as professionals we put in the ear plugs when that sort of stuff starts happening and get on with what we should professionally and that is concentrating on ourselves and our game and not getting unfocused by praise or criticism.
"You get praise through the media brilliantly as well but that can affect you just as much as the criticism so just don't listen to it, try to keep it at a minimum and keep focused and moving forward."
Joe Cole, another former England star who has struggled with injuries, played his first 90 minutes of the season for the Hammers in the Capital One Cup win over Burnley in midweek after recovering from a hamstring problem.
"It was a big boost for us as we never expected Joe to be out as long as he was," Allardyce added.
"It was a real blow after the way he started against Cardiff and then picked up what was considered only to be a slight hamstring problem before the game against Stoke.
"It's taken a lot longer than we thought. As long as we keep Joe fit he is a massive player for us and one we look forward to using and staying fit."