Brown's contract at the Memorial Ground, which expires at the end of the month, will not be renewed, with the player set for a second spell in Finland.
The former Fulham man, 24, has made more than 50 appearances for the Pirates, but has decided not to continue his spell with the League Two side.
McGhee told the club's official website: "Wayne decided on the back of the offer that we made him that he didn't think it was enough and he is going to be leaving the club in the new year.
"I think it is probably the right thing for him.
"He has been a great professional. He trains well every day and gives you 100 per cent every time he plays, and we wish him well."