Lord Herman Ouseley's comments follow criticism levelled at UEFA for what is considered to be their lenient response to racist chanting and violence aimed at England under-21 players during their match in Krusevac, Serbia, in October.
The match ended in a mass brawl involving players and coaching staff from both sides and, on Thursday, UEFA fined Serbia's FA 80,000 euros and ordered them to play one match behind closed doors.
Players from both sides were handed bans - including England duo Steve Caulker and Thomas Ince - and Lord Ouseley is another critic unhappy with the outcome.
Kick it Out has campaigned for equality and inclusion, and against racism in football, for almost 20 years and Ouseley told the BBC: "If necessary games should be stopped and sections of the crowd who are behaving badly should be shown the door."
When asked if football was slipping back towards the bad old days, he added: "It clearly is if you have your position undermined by people who are in powerful positions, and I am talking about the FA."
Peter Herbert, chairman of the Society of Black Lawyers, claims that UEFA's stance on racism is 'abundantly weak'.
Speaking to Sky Sports News, he said: "As far as Serbia is concerned, they don't even include the word racism or discrimination in their final report.
"It just beggars belief as far as we're concerned that they should take such an incident, which is not the first time the Serbian FA have been guilty of this in terms of the conduct of their fans, and actually after the second or third occasion give what is effectively a green light to this happening again."
Meanwhile, UEFA president Michel Platini is considering an appeal to increase the one-match spectator ban as well as the fine on Serbia.
The sanction was imposed by UEFA's independent control and disciplinary panel, while UEFA itself can appeal to increase the severity of the punishment.
UEFA sources said Platini has taken note of the sanction and will make a final decision once he has read the full file on his return from Japan next week.
Don't miss any live and exclusive football coverage this festive season. Click here to visit Sky Sports' Festive Calendar