The former Manchester United defender was shown a red card by Kevin Friend for a challenge on Charlie Adam, who six minutes earlier had scored Stoke's first goal in a 2-0 victory.
TV replays show Brown taking the ball and making minimal contact with his opponent in the 36th minute.
And Gallagher says Friend made an error, telling Sky Sports News: "I can keep this very short. We have to take it smack on the chin.
"The referee has lost concentration or not seen it. It is not a red card, that's all you can say.
"Brown's gone in hard but he's entitled to do that. As a refereeing body you have to take that smack on the chin."
And Gallagher also felt Phil Dowd made a mistake in failing to send off Everton's Kevin Mirallas following his lunge on Liverpool's Luis Suarez in the Merseyside derby at Goodison Park.
Gallagher added: "He should have been sent off. He's come at speed and it's high.
"The referee has got a body between him and Suarez and he hasn't seen it."