Two goals from Andy Carroll and Gary O'Neil's first in the Premier League for nearly four years put West Ham comfortably ahead, although Graham Dorrans reduced the arrears from the penalty spot with two minutes to go.
Youssouf Mulumbu's sending off in stoppage-time at the end of the match, which sees the midfielder miss the next three games, made it a day Clarke was keen to put behind him.
"We knew this was going to be a difficult game," Clarke said. "West Ham would see this as a big chance to get three points towards making themselves safe in the Premier League.
"In the first half, West Ham were better than us at the ugly side, winning the second balls and keeping the pressure on us.
"They did that well, and we didn't do it well enough. The goals we conceded are not like us, they were three disappointing goals to lose."
West Ham took the lead shortly after a brilliant save from Jussi Jaaskelainen to deny Romelu Lukaku, and Clarke felt that had luck gone his side's way the afternoon may have taken on a different complexion.
He added: "We started the game quite well, but there are small margins in this game. Lukaku's free-kick got a deflection and it was a good save, if that had gone in we were in front, and it becomes a totally different afternoon.
"But within a couple of minutes we lost a man at a corner, Andy Carroll heads it in and from there it was a very difficult afternoon."
While not criticising Mulumbu outright for his sending off, which came when the Congolese player lashed the ball at Gary O'Neil in frustration, Clarke was disappointed at losing the player for three games.
"The red card did not affect the game today, but it affects our group for the next three games and that's what was disappointing. It was just frustration, that was clear to see."