Football Association chairman Greg Dyke said on Wednesday that action had to be taken to safeguard the future of the England national team, claiming the influx of foreign players is hurting the game and that there are not enough chances for homegrown youngsters in the Premier League.
"We all have a responsibility to do our best to reverse this frightening trend because if we fail we will be letting English football down and we will be letting the nation down," Dyke said.
Brown, however, is unsure how much more he can do as manager of a League Two club in Southend.
He told Sky Sports News Radio's Football League Hour: "We are trying to improve the national team, but that's not my remit.
"I'm working for a chairman and a football club that I want to get promotion with and I want to be successful with to get my career back up and running.
"The national team, am I interested in it at the moment? I'm a million miles from it.
"Yes, we're still trying to produce players but the fact of the matter is, if they're not good enough, we're not going to improve as a national team. So how do we improve what we've got?
"We've got to nurture these guys, but it's a nature/nurture thing. If they're not good enough, the simple fact is we're not going to improve as a national team."
Brown warned that, until the standard of young homegrown players reaching professional clubs improves, the situation will not improve.
He added: "At Bolton Wanderers we brought in a lot of foreign youngsters, and we had English youngsters alongside them, and guaranteed they weren't as technically as gifted as [the foreign youngsters] and weren't as tactically as gifted as them.
"If you're good enough you play, if you not good enough you don't."
Former Hull City and Bradford City striker Dean Windass, however, believes the amount of foreign youngsters in the game is denying homegrown ones the chance to improve.
"It's tough for the young kids now," he said. "It was tough when I was a young kid and it's nowhere near as bad as it is now.
"All these kids just want a chance, and at the minute they're just not getting a chance.
"My lad is at a feeder club in Accrington. It's a great opportunity for him because they've not got the biggest squad, and it gives Josh a better opportunity of getting to the first team.
"But I remember when he went training at Man City and he said, 'dad, they've just paid £3million for a 16-year-old kid from Portugal'.
"I'm not taking anything away from the foreign lads because they are talented, and they improved my game when Benito Carbone and Dan Petrescu came to Bradford.
"Dennis Wise said it at Chelsea. When Vialli and people like that came in, it changed the way [the English players] lived their life, how they train and what they eat. It was a positive but now it's not just the footballers, there are a lot of foreign owners, too."
Preston striker Kevin Davies, meanwhile, who was capped by England in 2010, believes the standard of coaching has improved at academy level but warns more must be done in grassroots football.
"Watching my son play at academy level for a few years I think the standard is good and the coaching has improved," he said.
"The coaching could still get better at grassroots level because you have a lot of dad's just wanting to run a team so I don't think we cast the net far enough in terms of players, but the coaches at academies are scanning a lot of players now so I'm hopeful that in 10 years' time we'll have a lot better standard of young players coming through."
To hear the full thoughts of Brown, Windass and Davies, as well as those of Doncaster midfielder Dave Syers, Millwall loanee Shaun Derry, Plymouth defender Guy Branston and Bristol Rovers defender Michael Smith, download the latest Football League Hour.
Tune into Sky Sports News Radio every Thursday from 7pm for more exclusive interviews from the Football League.