A small group of Pompey supporters made their views clear to Barker as he prepared to face the press.
Barker insisted the fans were entitled to their views, but insisted his team were making progress.
He said: "We created more chances than we have for quite a while.
"Fair play to them defensively.
"They threw themselves in front of everything.
"The start wasn't ideal and then there's the penalty.
"It knocks the stuffing out of you when you get a decision like that.
"I don't think Daniel Alfei even knew the lad was near him.
"You get situations like that dozens of time in every game and I just can't understand it. That made it tough then.
"The third has been a real blow and we've then started to fight back."
Jed Wallace did end Portsmouth's three-game goal drought but it was not enough to deny Graham Alexander's side.
The home side's opener came from a throw on the right, Jamille Matt getting ahead of the Pompey defence and although his shot was saved by Trevor Carson, Jon Parkin was well placed to stroke home the rebound.
The lead was doubled 10 minutes later. Steven Schumacher swung over a cross and after Daniel Alfei was adjudged to have pushed Josh Morris, Antoni Sarcevic fired his spot kick straight down the middle.
Charlie Taylor could have extended the home side's lead 10 minutes after the break, but was denied by Carson after racing clear.
But the respite for Pompey was short-lived. Matt took advantage of confusion in a crowded box to stab home Fleetwood's third after another set-piece.
Pompey were not sunk however, and hit back in spectacular fashion through substitute Jed Wallace, whose dazzling run was crowned with a stinging shot to end the club's three-game goal drought.
But it was not enough to spark a fightback as Fleetwood substitute Mikael Mandron had a chance to make it four for Fleetwood, but his shot was blocked.