A clumsy challenge from Pablo Armero on Cameron Jerome just before the hour mark presented Mile Jedinak with an opportunity to crash home from the spot in Saturday's Premier League clash at Upton Park.
Allardyce admits that gifting the in-form visitors an advantage was always going to make life difficult for his side, with fortune not favouring the Hammers at present.
On the back of a third straight defeat, he told Sky Sports: "It's a stupid penalty, isn't it? The last two games, three penalties we have conceded - two were justified, one wasn't. Of course we haven't won a game here because of three penalties that we have given away.
"The fact of the matter is that however you set out your plans, you can't judge for things like that as a manager. He didn't need to really make the tackle in the position he was in, so it was a mistake.
"You know the way that Crystal Palace set their team out, they are waiting for something like that, they are waiting to capitalise on whatever errors you might make. They took full advantage of that.
"It was always going to be difficult to break them down because they are the best team defensively, I think, in the Premier League based on how they shape up and close the ball.
"Because they are in such good form as well in terms of results, beating teams like Everton and Chelsea, the game was going to be difficult in breaking them down.
"We gave it every effort we possibly could but we couldn't put the ball in the back of the net when we had the chance and the opportunity to do so.
"I still think Andy Carroll's is a penalty when he gets blocked off (by Jedinak in the first half) with no attempt to play the ball, so one goes for you and one goes against you and you end up with a disappointing 1-0 defeat."
Another setback leaves West Ham seven points above the relegation zone with three games to play and Allardyce has called on his side to ensure that they end the campaign as positively as possible and do not allow themselves to be dragged back into a basement battle.
He added: "What we want to do is, we don't want to lose any more football matches.
"We have a tough run-in with West Brom away, Tottenham at home and Manchester City away.
"We have got to try and get as many points as we can out of the next three games and try to finish the season with as many as we can get, which is more than 37. That is the object of the exercise of the next three games."