The keeper was shown a red card after bringing down Borja Perez in the box on the stroke of half-time, from which Liam Kelly fired the visitors in front from the penalty spot.
Kelly added a second four minutes from time and manager Brown said: "The first goal was a defining moment and it became difficult for us.
"However, in the second half you would have thought that we had the extra man because we were dominant, although Cammy Bell only had one real shot to worry about, the one from Josh Magennis that came off the crossbar.
"It's a major disappointment to suffer a home defeat, especially after our performance on Tuesday.
"But I think there was some reaction to Tuesday night, and we had no experienced players to freshen up the team, with seven players out injured.
"Particularly in the early part of the game, you could see we were suffering, but we got a second wind in the second half until we lost the second goal."