New Seagulls manager Sami Hyypia revealed on Friday that Lee, his former colleague at Liverpool, was to join his backroom staff from Bolton where he had been working with the club’s academy.
However, the 55-year-old has now changed his mind and decided to move to Southampton where he has been offered a job working under the Premier League club’s new boss Ronald Koeman.
Brighton chief executive Paul Barber told his club's official website: "It is extremely disappointing for the club, Sami Hyypia and those of us who worked hard to bring him here, that Sammy Lee has now reversed his decision after agreeing to become assistant manager.
"Nobody at our club or Bolton Wanderers, was aware of any interest from Southampton until several hours after we announced Sammy's appointment as our assistant manager. It's certainly not a situation any of us have come across before.
"Having already agreed his contract and terms with us to start work on Monday, we could have insisted on Sammy serving a notice period, but we all feel it would have been counterproductive to do so; and while it's incredibly frustrating for us we wish Sammy well.
"Our focus now is on working with Sami to help him recruit his assistant - while also supporting him to bring in the players he needs to improve the squad for the new season."
Bolton have also confirmed Lee's departure to Southampton from his role as Head of Academy Coaching at the club.