Gary Whybrow, 31, Sam Parsons, 24, and Peter Ditchman, 52, were initially charged with using the language at Tottenham matches last autumn.
However the Crown Prosecution Service said the words could not legally be counted as "threatening, abusive or insulting" in the circumstances.
Baljit Ubhey from the CPS said: "In accordance with our duty to keep all cases under review we have conducted a senior level review of this case.
"It has now been concluded that, according to the Code for Crown Prosecutors, there is insufficient evidence to provide a realistic prospect of conviction, and that the cases against Peter Ditchman, Gary Whybrow and Sam Parsons should be discontinued.
"In considering whether a criminal offence could be proved we have to look objectively at the words used, and the context in which they were used.
"As part of the review, the context of the use of the words alleged in this case was reconsidered, and we have decided that, although the same words used in other contexts could in theory satisfy the criteria for 'threatening, abusive or insulting', it is unlikely that a court would find that they were in the context of the three particular cases in question.
"We have therefore concluded that there is insufficient evidence for a realistic prospect of conviction. This decision has no bearing on any other cases that may be brought to our attention and all cases will be considered on their own facts and merits."
A statement from Tottenham read: "The Club welcomes the recognition of the importance of context. We have always maintained that our fans do not use the Y-word with any intent to offend.
"As with all cases where fans are arrested for football-related offences, we were obliged to issue bans and the Club can confirm that these supporters' bans have been rescinded with immediate effect and refunds will be made for matches missed on season tickets, as per Club policy."
The trio were accused of a public order offence for using the word. Police had previously warned football fans not to use the word, which is used to refer to Tottenham fans and is regularly used in football chants.
Meanwhile, Chelsea have promised to take the strongest possible action and press for criminal convictions if anyone engages in anti-Semitic behaviour during Saturday's Premier League clash with Tottenham at Stamford Bridge.