The Potters saw their winless run stretched to six games on Saturday, with a 3-1 defeat leaving them just four points clear of the Premier League relegation zone.
Pulis believes Stoke have not been getting the rub of the green over recent weeks, and feels that was again the case against Villa.
Paul Lambert's side survived two loud penalty shouts at the Britannia Stadium, and saw Matthew Lowton turn the match in their favour late on with a spectacular 35-yard volley.
Pulis told Sky Sports afterwards: "We are disappointed. I thought we started well, but we gave such a soft goal away. It was a real soft goal and, to be honest, it looked as though it drained a few of the lads' confidence.
"The crowd were good today, they turned up to get behind us and really tried to push us on.
"In the second half, Villa had the better chances and the better opportunities on the break, I thought they broke really well and looked really dangerous. We got the goal and you think we will go on and win it, and then Lowton's goal is one in a million."
He added: "We could have had two penalties. The first one, it's definitely hit his (Nathan Baker's) hand and his hand is away from his body. The second one, (Ron) Vlaar, when the ball hits him, it stops it going to Kenwyne Jones.
"The linesman has got a great view of the first incident and Mark Clattenburg has got a great view of the second one. I don't know what is handball -sometimes they give it, sometimes they don't. But certainly the breaks aren't going for us."
Stoke now have six games in which to save their season and Pulis insists there remains a belief within the camp that they can preserve their top-flight status.
He said: "There are six games to go, we have got to get to 40 points, and it's tough. The run we are having at the moment, we are certainly not getting the breaks. It's a test of everybody's character around the place.
"You have got to believe. We have as much as anybody else. But the players are the ones that you have got to give the belief to and they have got to be the ones that have got to step up to the mark."