The West Ham manager made his comments following the decision by the FA to rescind the red cards shown to his own striker Carlton Cole and Everton's Darron Gibson at Upton Park last week.
Allardyce believes there should be a larger pool of referees to cope with a congested fixture schedule over Christmas and New Year and believes tiredness can lead to officials making the wrong call.
He said: "This is a difficult period for referees. Why do they always seem to make more wrong decisions now?
"It's probably because they are overworked and they are fatigued and the system has to be looked at.
"Referees are travelling all over the country and out in Europe and have been since the start of the season.
"There's not enough of them. They try their very best to get things consistently right. But it happens every year around this time of year.
"We had a couple in our last game and there have been many throughout football over the Christmas period.
"Fatigue is everything in terms of decision making. Once it kicks in you lose the ability to make those decisions correctly.
"They need to ease the load on the referees we have. The game needs to move on and look at recruiting better refs from all avenues."