Spurs won the game 3-1 but the main talking point after it was not matters on the pitch but the anti-Semetic abuse coming from the away support.
Hammers boss Allardyce insists though that he was concentrating on the football.
"I didn't hear it," he said. "I don't hear what the fans say or do when I'm concentrating as a manager on a game of football.
"They shouldn't be doing things like that, it is the least of my worries at the minute isn't it? What do you want me to say? If I didn't hear it I can't condemn it can I?
"I will wait and look at it myself and make a comment once I have listened to what they have said.
"I don't want to be a political animal, I'm here to talk about football and not what fans are saying and singing."
On the match itself Allardyce admitted he was not happy with his team.
"You can accept defeat when you know your players have played their best and we haven't done that today," he said.
"I don't accept the way my players have played today and I want them to bounce back."
Just a matter of days after Ashley Mills was stabbed in an anti-Semitic attack before Tottenham's game in Rome, West Ham supporters were heard to sing "Viva Lazio", "Can we stab you every week?" and hissed on several occasions, apparently mocking the mass execution of Jews during the Second World War.
Tottenham manager Andre Villas-Boas said he did not want the issue to detract attention from what was his side's best home performance of the season.
"I prefer not to mar the performance with a situation like this," the Spurs manager said.
"You know the animosity there is between Tottenham and West Ham and as long as it doesn't reach stupidity it is a great, great rivalry of two London clubs.
"It would be extremely unfair for me to mar the performance of the players but understanding that a couple of situations are avoidable but we can't decipher the true meaning of what they were saying."