The visitors took the lead early on through Andy Carroll but struggled to increase their advantage and were punished through two goals from Peter Odemwingie and one from Marko Arnautovic.
West Ham thought they might have had a penalty in the second half when Marc Muniesa appeared to handle in the box but referee Craig Pawson waved away Carroll's protests.
Speaking to Sky Sports after the game, Allardyce was convinced Pawson had got it wrong.
He said: "I can't control what the referee does but I can with our defending and that was poor at times today.
"We had a back four intact and none of them did anything to stop Stoke's players getting through and scoring.
"The critical decision didn't go for us and straight away it's a pen, we can all see it.
"Andy Carroll was going mad. You do hope referees don't make those decisions but that was certainly a penalty."
The West Ham boss's next target is to secure survival and then decide which players will be in his plans for next season.
He added: "Now, and as quickly as possible, we need to get enough points on the board.
"Things needs to be sorted by securing our safety and then we can look at the players that are out of contract.
"It's a big disappointment today but you always expect Stoke to come back.
"One of the telling factors was the wind. We penned them back in the second half until they scored on the break. It was a blatant handball though by the substitute and his arm knocks the ball away from Carroll and the referee chooses not to blow his whistle.
"Defending wise, we have no excuse though."
West Ham face Manchester United next weekend in the Premier League.