Barker took over at Fratton Park just a fortnight after he was axed by the Red Devils.
He leaves behind some highly-rated players such as Joe Walsh, Nicky Adams and top scorer Billy Clarke.
But Barker said: "Will I go back? No. These players here deserve the opportunity to stay here and do the right things.
"If they do the right things they will get the opportunity to stay.
"Shaun Cooper was with me at Crawley and Trevor Carson came through at Bury so there are two players here I have worked with before."