The centre-back saw red in the 25th minute after bundling into Stewart Downing as he burst clear, with Andy Carroll seeing his resulting free-kick superbly tipped over by the excellent Hugo Lloris.
Carroll got his head to the resulting corner and saw it deflect home off the head of the unfortunate Harry Kane, before Downing scored his first goal for the Hammers before the break as Tottenham's wall parted for his free-kick.
"I don't think we were outfought," Sherwood said. "The game changed on the red card.
"I thought before that we were comfortable and created the best chance of the game when Ade had a chance.
"It was always going to be a tough place to come, and when we go down to 10 men and they score a minute after it is going to be a difficult afternoon.
"Then, obviously, the free-kick goes straight through the middle of our wall, which isn't ideal, and then you've got the mountain to climb."
Sherwood had no complaints about Kaboul's sending off, nor was he particularly angry with Paulinho and Emmanuel Adebayor despite the pair leaving an embarrassingly large gap in the wall that Downing duly exploited.
"It wasn't great but I spoke to them in there," Sherwood said. "I have pointed it out to them and they don't need me to point it out, really.
"They know it is an error and it obviously hurts a lot more when you see 2-0 rather than getting it in the stomach, when it hurts for 30 seconds."
Not even the inspired display of Lloris could keep Downing's free-kick out, although the France goalkeeper produced a string of wonderful saves to stop Spurs receiving an embarrassing thumping.
"Lloris was top draw," Sherwood said. "He has been top draw whenever we have played this season.
"I heard people talking after the Everton game when he got a bang on the head that he is not the same goalie any more. That is a load of crap.
"People don't know what they are talking about. The guy is a consistent performer - he has performed like that every single week.
"If he makes a mistake, it is an honest mistake and he doesn't make many."