The hosts appeared to be heading for three points at the Liberty Stadium on Sunday, until Palace levelled late on through a controversial penalty.
Dutch keeper Vorm charged off his line in a bid to stop Glenn Murray but the Palace striker got to the ball first and was then brought down by Chico Flores.
"I came out, I had to clear it," said the Swans shot-stopper. "But he was a bit quicker with his legs and then it was a foul. But in my opinion, it was outside of the box.
"The referee was quite far (away). Maybe the linesman could see something. Decisions are made and you have to accept them. Penalty, 10 men, so I think in the end we are happy with the point. These kinds of games, we need to win them, especially at home.
"(The draw) was more because of us than because of them. We couldn't win the second ball, we couldn't play the ball out. It was very tough. You could also feel we played a lot of games, especially the Thursday games with the travel.
"You could tell we were tired at the end. It's hard but if you want to grow as a team, you have to cope with that. You can't do anything about it. You have to give everything but obviously it's easier if you only have one game a week.
"It's very tight down there now but I'm 100 per cent sure we don't have to worry. The next game is West Brom. These are the games you need to win. No games next week, so we will be fully fit for West Brom."