It was Rovers' third successive loss and left them 23rd in the League Two table with only four wins from their 22 matches this season.
The Pirates were 4-1 down after only 33 minutes at Bootham Crescent and the result was the final straw for the board.
Chairman Nick Higgs said: "We had hoped that the appointment of Mark McGhee would give the club some stability and long term progress, but unfortunately that has not been the case.
"We have decided that we can no longer continue with the current situation and had to make the difficult decision to part company with Mark. We decided we had to act now before it was too late."
McGhee, 55, only took charge at the Memorial Stadium on 18 January following the departure of Paul Buckle.
The Scot had previously managed at Reading, Leicester, Wolves, Millwall, Brighton, Motherwell and Aberdeen.