The 49-year-old Italian took charge of the Turkish club in September last year on a three-year contract after leaving City at the end of the 2012/13 season.
He guided Galatasaray into second place in the Super Lig behind Fenerbahce and also won the Turkish Cup, but they exited the UEFA Champions League at the last-16 stage after losing to Chelsea.
"Our relationship with coach Roberto Mancini and his contract of employment, which was signed on October 1, 2013, has been terminated by mutual agreement," read a statement on Galatasaray's official website.
"We wholeheartedly thank (Mancini) for his success and achievements and wish him happiness in his next mission."
There were constant rumours in Turkey about Mancini's future in recent months but he insisted only two days ago that he would be staying with Galatasaray.
However, a dispute regarding the club's future aims appears to be the reason behind his sudden exit.
"As a coach I understand the demands of the club," said Mancini. "However, when I accepted the coaching post, Gala's aims were different.
"In the past eight months, despite enduring a challenging start, I have done my job pushed by passion and by hunger which has resulted in the team qualifying for the Champions League, lifting the Turkish Cup and finishing second in the league.
"It's with sadness that I leave Istanbul and the club's fantastic supporters."
Mancini has been linked with a return to his former club Lazio in Italy and his odds to become the next manager of Southampton have also shortened following his departure from the Istanbul-based club.
Saints executive director Les Reed revealed on Tuesday that the club are close to announcing a high-calibre managerial appointment to replace Mauricio Pochettino.