The Potters took a 2-0 first half lead at the Liberty Stadium thanks to goals from Jon Walters and Stephen Ireland.
But the Swans fought back after the break, with a Wilfried Bony brace and goal from substitute Nathan Dyer giving the hosts the lead with four minutes remaining.
Wayne Routledge was then adjudged to have handled in the box and substitute Charlie Adam notched the resulting spot-kick to lift the Potters out of the Premier League drop zone.
"Given our history, in recent terms, we don't usually get them," Hughes told Sky Sports.
"Listen we take the break, there was contact.
"I think Steven (Nzonzi) tried to flick the ball on, he was right close to the goal, and who's to say that if the lad hadn't got in the way, it might have gone in anyway?
"So I can understand Swansea being a little bit aggrieved but we've had situations like that ourselves."