Charlie Adam put the hosts ahead on the half-hour mark before Brown's dismissal nine minutes before the break.
Steven N'Zonzi secured the points for Hughes's men late on but all the talk afterwards was about Brown's sending off.
Speaking to Sky Sports after the game, Hughes thought his team were on top despite the decision.
Hughes said: "It was all about the three points today. We have disappointed in very few games this season and today we deserved it.
"People might say the red card was pivotal but we were 1-0 up. I always felt we were in control and we scored two really good goals today.
"The referee obviously thought the challenge was reckless and that's how he interpreted it. You can't go in like that in this day and age."
Sunderland were furious later on in the game when they thought Stoke City keeper Asmir Begovic caught Steven Fletcher when through on goal. For Hughes, though, that was not the case.
Hughes added: "Asmir cleared the ball and moved his foot away if anything.
"Other than that in open play they didn't really create anything and I was happy with how we closed the game out in the last ten minutes."