While quick to praise Spurs' Gareth Bale, who won a pulsating match with a superb last-minute winner from long range, Allardyce felt it could all have been different.
The Hammers had come from 1-0 down to lead 2-1 and substitute Taylor was then put through on goal, but Hugo Lloris rushed out to save well and Spurs equalised soon afterwards, before Bale's wonder goal clinched it.
Allardyce told Sky Sports: "We've done a terrific job and made a terrific go of it. Jussi (Jaaskelainen, goalkeeper) played so well.
"It was at a crucial stage when Matt Taylor got through one-on-one. The 'keeper made a good save, but we want to be scoring that and, at 3-1, there's no way back for them.
"In the end, we've been beaten by Gareth Bale. He continues to produce world-class strikes. You'd think he's run out, but that was probably the best of the lot.
"The way he hit it, there's not a lot you can do and, at that stage, there's not much time to come back.
"If you go in too tight on Bale, he skips past you and, if you make him hit it from 30 yards out, it's got to be a hell of a strike to score. But, as soon as he hit it, I went, 'Oh no'.
"It's a very disappointing end to one of our best performances here against a top side. We caused them a lot of problems.
"We went 1-0 down to their first shot on target, got 2-1 up, had a chance to get 3-1 up, and then paid the price."
Allardyce was not happy that Tottenham midfielder Mousa Dembele, who was booked in the first half, was not given a second yellow.
"Consider the consistent bookings that Howard (Webb) gave our players," said Allardyce. "But, after getting booked, Dembele fouled (Mohamed) Diame and the ref let it go.
"And, in the second half, he (Dembele) pulls Joe Cole back and then kicks the ball away. If he does that, he's got to send him off, that's what we're told, so I'm really disappointed.
"At 1-1, that makes a huge difference. I'll put it in my report and see if the boys agree with me. If they don't, I'd like to know why not."