After a goalless first half, the Gunners fell behind when substitute Michu found the back of the net, but Wenger's men soon levelled through their own super-sub Lukas Podolski, before Kieran Gibbs' stunning volley looked to have sealed the win.
But Swansea hit back, with Danny Graham conjuring a clever far-post finish, which saw the third round tie end 2-2 with an Emirates replay now required.
Speaking to the club's official website, a frustrated Wenger said: "We should have won this game. I felt the first half was a bit locked and one-paced, as much as the second half became much more open.
"I think in the second half we created many chances and suddenly found ourselves 1-0 down. We have shown character and quality.
"I felt when we conceded the corner [that led to the second goal], we lacked a little bit [of] calmness and focus. It's frustrating to concede a goal just like that on a corner where we couldn't clear the ball before the corner.
"Even on the corner, we were first to the ball and didn't win the ball. We have shown quality and heart and it's quite good if we can repeat that."
The Frenchman cited his frustration at having to replay the third round tie, also having a previously postponed game to catch up on.
"I am frustrated because I wouldn't want one, but if that's the choice between going out or staying in the hat, I take the replay. They have many games as well. We have West Ham on January 23," he said.
The second half saw a penalty appeal waved away by Howard Webb, as Aaron Ramsey tripped over his own feet in the box with Dwight Tiendalli in pursuit.
The Arsenal manager, who disagreed with Webb's decision, said: "It was one and the referee saw it. You know why he didn't give it? Because he thought the guy had not done it on purpose. It was accidental but he didn't trip himself.
"He was caught clearly by the leg of the Swansea player. I don't know who it was. He had a hesitation, the referee, and in the end he didn't give it."