Bradley Dack came within inches of giving his side the lead in the 15th minute - his finish beat Bartosz Bialkowski but came back off the post.
But just minutes before the break Callum McGregor gave his side the lead, twisting and turning inside the Gills' defence before seeing his shot deflect off skipper Adam Barrett.
Within two minutes of the restart, Gillingham were level - the County defence failed to deal with Charlie Lee's long throw-in and the ball eventually fell to Dack who bundled the ball home.
Gillingham were in control, but completely against the run of play County re-took the lead through McGregor before a wonderful solo goal from Jack Grealish sealed an away day to forget for Taylor.
"I think the performance was okay," he said. "We created enough chances.
"I'm very disappointed with the finishing and very disappointed with the goals conceded, because they were poor goals. The third goal, we made their player look like George Best.
"We created a lot of chances; I felt it was amazing that we were 1-0 down at half-time. It was great that we got one at the start of the second half, but then we had to not give anything away again.
"It wasn't like our performance against Rotherham. We all admitted that our performance against Rotherham wasn't good enough.
"This performance was good enough, but it was individual errors that cost us. There was more about us, so that's an improvement, but we can't afford to give cheap goals away."