Charlie Lee, who struck seven minutes after coming off the bench, sealed the win for Gillingham with two minutes to go.
While Gills boss Peter Taylor said the game could have gone either way, Gregory was certain his side had dominated.
"It's difficult for me to get angry with my team, because it's normally a difficult place to get anything from," Gregory said.
"We've totally dominated the game, our chances were far more clear-cut.
"We've missed some guilt-edged chances that should have gone in and we should be talking about a three or four-nil away victory.
"They had one chance in the final couple of minutes and they nicked it.
"You don't need to be particularly skilful to put the ball in the box from a set-piece and that's what happened for the goal.
"It was a travesty really, but we've only got ourselves to blame in many respects.
"Our goalkeeper had one save he had to make in the second half, which he did brilliantly for, but he spent long periods of the game as a spectator.
"I expected us to be put under pressure more than we were, but we dealt with everything.
"That was my first away defeat, which is hard to take at the minute, because Gillingham were there for the taking."