The 22-year-old centre back has signed a two-year deal with Mark Yates' men and returns to League Two having been relegated with Aldershot last season.
"I'm buzzing to have this opportunity," Brown told the club's official website.
"It's a good footballing club and a well run one too so I can't wait to get started. The club had a very good clean sheet record last season and I would hope to build upon that because I put my body on the line when it's needed.
"I want to win something and the club's league standings over the past few years speaks for itself."
Brown, a Wales Under-21 international, is reunited with former team-mates Jermaine McGlashan and Jason Taylor at the Abbey Business Stadium.