The chanting occurred during England's recent match in San Marino and was a reaction to Rio Ferdinand's decision to withdraw from the squad, so he could stick with a pre-planned fitness regimen.
It has even been suggested that one of the songs had racist overtones and anti-racism group FARE reported England to FIFA on Thursday.
Ouseley told BBC Radio 5 Live: "Whether it's racist or not, it's certainly unacceptable. It's vile and it shouldn't be part of sport. Something needs to be done about it.
"These are the supporters of the England national team who are travelling abroad and singing songs like that.
"What message does it send out about the type of people we are and who we represent?
"The Football Association should be taking a stance on this about the people it wants supporting the England team, the image it wants to send abroad.
"I've already contacted the chairman of the Football Association and said this has to be looked at, investigated and dealt with. Do you want to be having an army of fans who call themselves the England fans travelling abroad, being abusive to their own players like that, or indeed other people?
"FIFA will determine (whether it is racist) but clearly we can take a stance on that. We have policies on anti-racism, homophobia, and all other forms of unacceptable behaviour, why are we so quiet about it?"
Regarding the chanting, FIFA said in a statement: "We can confirm that FIFA has been contacted by FARE regarding the World Cup qualifier match between San Marino and England last Friday.
"FIFA will now analyse the content of the documents and next steps will be determined in due course."