The Shrimpers failed to add to Ryan Leonard's strike with Gavin Tomlin seeing his goal disallowed in the final five minutes and Barry Corr having a shot cleared off the line in stoppage time.
Brown was saddened the way his team played and has told his team their performance was not good enough.
"I was bitterly disappointed", he said. "I didn't think we attempted to play any kind of football. We were playing for scraps, little bits and second balls and today it was not good enough.
"Certainly the two goals we conceded defensively, the first goal from a long throw and the second we had about eight in our box and they still managed to score. When you defend like that you don't deserve to win any matches.
"When you go out there you have to go out and give your best as an individual and we were found wanting in the battle which has disappointed me.
"Every time I've seen players cross the white line for Southend they've battled and they've scrapped and deserved to get what they got.
"I've just said to the boys in there where you finish in the season is where you belong whether you're a top three team or a bottom three team."