Williams rejected the deal offered to him and leaves the club as a free agent after three seasons in north Essex.
"Ben wants to play in the Championship and feels that now is his best opportunity," manager John Ward told the club's official website.
"We gave him quite a lot of leeway in terms of his new contract, three weeks more than we had to, but we have to respect the decision he's made.
"I've said all along though, that if he doesn't sign, we will give Mark Cousins his go.
"The supporters know as well as I do that we have got a very good keeper, more than capable in fact, and now it is his time."
