Andre Marriner, who takes charge of Saints' home clash against Newcastle on Saturday, accidentally sent of Kieran Gibbs instead of Alex Oxlade-Chamberlain in Arsenal's 6-0 defeat at Stamford Bridge last weekend.
Marriner was forced to issue an apology for his mistake but was not reprimanded by the Football Association.
But Fonte believes referees need to be banned in the future if such glaring errors are made during games.
When asked if referees should be dropped, Fonte said: "It is a question for debate, personally my opinion is yes. If you make mistakes as players we are punished.
"We get dropped. I think obviously some might argue whether that it is good or not but I personally think they should get dropped as well and don't referee for a couple of weeks to realise what went wrong.
"I just think when the mistake is very, very bad you need to stop and think 'What can I do better?'
"We are aware of the situation with the referee but it won't be a big issue. He has a tough job to do but I am sure he will want to be perfect on Saturday."