The Newcastle boss has served a seven-game ban - three from the stadium and four from the touchline - after head-butting Hull City midfielder David Meyler in the 4-1 win at the KC Stadium at the start of March.
During Pardew's absence, Newcastle have endured a miserable run of form, losing six and winning just one of their last seven games.
March's last-gasp 1-0 win at home to Crystal Palace was the last time his side collected three points in the Premier League, and Pardew says he was not expecting such a slump in results.
"It is hard to believe really, when we beat Palace (in March) that we would end up like this," he told the Evening Chronicle.
Wilfried Bony's stoppage-time penalty grabbed Swansea a 2-1 win in the final game of Pardew's ban, making it five straight Premier League defeats.
Pardew said: "I'm absolutely gutted with events after the last-minute goal.
"I didn't think we deserved it. Unfortunately we are going to have to wear it."
Meanwhile, French team-mates Mathieu Debuchy and Loic Remy made returns from their injuries earlier than expected against Swansea after Papiss Cisse and Luuk de Jong were forced off in the first half.
Speaking of the duo, Pardew said: "Physically they weren't at their sharpest. You wouldn't expect them to be.
"They came on and that was the maximum they could have played. I had no problem with those two."