The midfielder, 20, arrived at Priestfield in mid-February and has made a total of eight appearances for the League Two club, although Saturday's late cameo off the bench was his only run-out in the last four weeks.
And manager Andy Hessenthaler has now decided not to extend Lee's stay as he prepares to welcome back a wealth of midfield options after injury.
"With the likes of Curtis Weston, Charlie Lee, Danny Jackman and Jack Payne coming back, we decided to let Olly go back to West Ham," Hessenthaler told the club's official website.
"He was in the side and then came out of it, it was probably a bit frustrating for him. But he was a pleasure to work with and we wish him well.
"It didn't work out for him, he'll tell you that himself. He just didn't get going, in the Cheltenham game (in mid-March) he did ever so well but in general he perhaps didn't get going in his spell with us."
