Rangel was furious with the official after he awarded a stoppage-time penalty to the visitors, which Charlie Adam converted.
Jonathan Walters and Stephen Ireland had put Stoke 2-0 up but goals from Wilfried Bony (2) and Nathan Dyer looked to have secured all three points for the home side before Madley's controversial intervention.
Madley incensed the home players and fans when signalling that Wayne Routledge had handled in the area six minutes into time added on, despite little, if any, protest from Stoke players.
"I am obviously disappointed by that decision but we can't talk much about the referees," said Rangel.
"We are always the ones who get fined but hopefully in this case it won't be the players who get punished but it will be the referee as we have done nothing wrong.
"When referees make mistakes they should be the ones to get punished."
Visiting goalkeeper Asmir Begovic, who was in the Swansea penalty area when the incident happened - looking to score his second goal in eight days - said: "I saw a scramble happen but I am not sure if I saw a handball."
Swansea manager Michael Laudrup said: "In every game there are talking points over decisions, but when I saw it again, or even out there, there are seven Stoke players in the box and none were asking for a penalty, apart from Peter Crouch who who was appealing for a corner.
"So there was only one man in the stadium who thought it was a penalty, and unfortunately that was the referee."