Yid is a term for Jew which is often considered derogatory, but fans of the north London club chant the word as an act of defiance against those who taunt them because of their links with the Jewish community.
Bernstein, who is Jewish, finds the word offensive but can understand why Tottenham supporters have adopted it "almost as a badge of honour" and claims their right should remain.
Scotland Yard said supporters of Tottenham and West Ham could be arrested if they used the word during last Sunday's London derby at White Hart Lane, and one Tottenham fan was held on suspicion of committing a section five public order offence at half-time in the stadium's East Stand.
However, many Spurs supporters defied police advice and chanted "Yid army" before and during the match.
"If two or three people do something, you can prosecute them. If a number of thousands do it, it gets increasingly difficult," Bernstein said.
"Then you get into the question of what penalties should be incurred. Should a stand be closed down? Should a ground be closed down?
"That's where you end up, and I'm not saying that should be the case here because I do feel this is something that's done in a non-malicious way. There are extenuating circumstances.
"If the word is used in the way it is used by the crowds, in a sympathetic, inclusive way, almost as a badge of honour, I suppose I would be against prosecutions and I would tend to agree with the Prime Minister when he voiced similar views.
"When words like that are used maliciously it's a different matter altogether."
Bernstein, who stepped down as FA chairman this year after turning 70, said he has "agonised" over the use of the word.
"It's very offensive to many, including myself," he said on BBC Radio Five Live.
"Would I rather it wasn't used? Of course.
"So I suppose the question is: if an offensive word is used in what's meant to be an inoffensive way, does it make it any less hurtful?
"My view would be that I wish that it wasn't used. It would be better if it wasn't. It does upset people; it certainly upsets me.
"But in this particular case it's a rather special set of circumstances."