Many Tottenham fans, both Jewish and non-Jewish, call themselves the "Yid army" and Spurs players as "Yiddos".
The Football Association recently issued new guidelines and suggested that any fans using the term could face criminal action but the Prime Minister appears to disagree with that stance.
"There's a difference between Spurs fans self-describing themselves as Yids and someone calling someone a Yid as an insult," Cameron told the Jewish Chronicle.
"You have to be motivated by hate. Hate speech should be prosecuted - but only when it's motivated by hate."
However, Herbert, chairman of The Society of Black Lawyers, does not agree and believes the law should apply within football grounds.
"David Cameron, luckily, is not in charge of the Crown Prosecution Service or the Metropolitan Police, therefore these decisions are not his to make," he said.
"These comments are like saying that I actually condone anti-semitism or racism. It's no better than that, and it doesn't get any better just because you're the Prime Minister saying it.
"What we're concerned about is the impact on the victim and that is an offence under the Public Order Act 1986, whatever Mr Cameron thinks.
"So we really have to understand the fact that people in the Jewish community, not all of them, and certainly many people in the wider community black and white, have a real problem being on the receiving end of these comments."
"Football is a part of society and not separate from it, so the Prime Minister has to really think what he is saying because he legitimises anti-semitism and that is a sad thing for any parliamentarian to do.
"You cannot have people breaking the criminal law on Saturday afternoon and saying it is okay because we have a badge of honour."