The visitors took the lead on six minutes after former Yeovil goalkeeper Steve Mildenhall failed to collect Andy Welsh's free-kick from the right flank. Southend were unable to clear the ball and centre-back Brown volleyed the ball into the net from six yards.
The Shrimpers chased for an equaliser but saw Adam Barrett and Lee Barnard head off target before Glovers goalkeeper Josh Wagenaar dived bravely at Kevin Betsy's feet as he prepared to shoot in first-half stoppage-time.
Yeovil were reduced to 10 men seven minutes after the break when Aidan Downes was shown a straight red card for a two-footed challenge on Johnny Herd.
Wagenaar kept his side in front by diving at the feet of substitute Theo Robinson before denying Franck Moussa and Jean-Francois Christophe with smart saves.
The Canadian then produced a great save to foil Belgian midfielder Moussa and Glovers skipper Terrell Forbes headed over his own bar as Town hung on for the win.