The Hammers missed a penalty midway through the first half when Mark Noble sent his spot-kick past the goal moments after Kyle Naughton had been dismissed for handling the ball.
But having had James Collins sent off for a second booking later in the match, the game appeared to be heading for stalemate when Spurs scored a late goal through Eric Dier.
“I was sat on the bench thinking ‘this is just our luck’,” Cole told Sky Sports.
“I thought the boys put in a massive shift and we were unlucky. I thought we’d see it out, be happy with a point and move on to the next game.
“It was a sucker-punch. I don’t think we deserved that. The boys are so disappointed. Everyone is down in the changing rooms.
“We’ve got to pick ourselves up, get ourselves out of the zone we’re in – we don’t want to be losing to Tottenham on the opening day.
“We should have made more of the one man advantage, and could have scored the penalty and set the tone for the rest of the game.
“We need to get this out of our system now. We’re going to fight again but we just didn’t have the finishing touches at the end.”