The behaviour of fans has blighted Serbian home games in the past, with their Football Association fined by FIFA for "crowd disturbances" during their qualifier against Wales in September last year.
There was further controversy in the Under-21s game with England in Krusevac in October, when Danny Rose alleged he had been the victim of racist chants and the match ended in ugly scenes as players and officials rowed.
But Tosic says Serbia's concentration is on the game ahead of their Group clash with Scotland on Tuesday night.
The CSKA Moscow player told Sky Sports: "We don't need to think about that because I think we have all turned to the other side and we're not thinking any more about that.
"We're trying to focus just on football and the supporters also. They're fantastic every game."
Serbia suffered a 2-0 defeat to fierce rivals Croatia in Zagreb last week, with travelling fans banned from attending the match, and Tosic expects the home crowd to be fired up on Tuesday following that loss.
He added: "The fans are nervous, they want a result. Sometimes they can turn against us."