Players' leader Clarke Carlisle has called on Tottenham to stop their fans using the word 'Yid' in chants.
The Prime Minister has even entered the row about the use of the term after the Football Association warned supporters that use of such words could result in either a banning order or even criminal charges.
David Cameron said Spurs fans - who use the term as an act of defiance after anti-Jewish abuse in the past from rival supporters - should not face prosecution.
But Carlisle, chairman of the Professional Footballers' Association (PFA), said he agreed with the comedian David Baddiel who argued that there would be an outcry if a team with old roots in the black community appropriate racist language.
Carlisle said: "Do they have a right to appropriate that term when it would be indescribably offensive to anyone else?
"David Baddiel says that's how it feels as a Jewish man going to Tottenham and hearing them chant that. If it is highly offensive to him then I think Spurs have to take that on board, because he will not be the only person.
"It is not for them to appropriate a derogatory offensive term that was used to belittle a whole section of society in a terrible era."
Carlisle, speaking at a book signing of his autobiography 'You Don't Know Me But... A Footballer's Life', pointed out that such chants breached the law.
He added: "I don't feel it's right. Spurs fans may not intend for it to be offensive but it will be perceived to be offensive by a section of the community and the law states that's not allowed - it's not even my personal opinion, that's what the law states."
Tottenham responded to the FA warning by announcing they would send a questionnaire to all season ticket holders asking if the practice should stop.
Fans reacted defiantly to the FA's statement on Saturday as they chanted "Yid Army" and "We'll sing what we want" throughout the 2-0 win over Norwich.
Cameron told The Jewish Chronicle: "There's a difference between Spurs fans self-describing themselves as Yids and someone calling someone a Yid as an insult.
"You have to be motivated by hate. Hate speech should be prosecuted - but only when it's motivated by hate."