Bywater, who signed on-loan from Millwall on Friday, enjoyed a dream debut as he saved Kieffer Moore's penalty after 17 minutes.
And goals in either half from Joe Martin and Danny Kedwell gave the Kent side a hard-fought victory over the Glovers.
"I'm pleased with that - I think we deserved the three points," said Taylor. "It wasn't a fantastic match, I think it was two teams trying hard not to let the other one in.
"I was very pleased with Stephen. The way he took his first cross was good, and that gave everybody confidence. He came in the building on Friday, and that gave everybody a lift. He's desperate to play to football.
"I think it was definitely a penalty for Yeovil. It was a long ball from their goalkeeper, and we didn't deal with it.
"It was a great free-kick from Martin. It was a nice distance for him; he can get it up and down from there perfectly. That was exactly the same as his goal against Peterborough last season. He needed that because he wasn't having a particularly good time."
Bywater saved Moore's tame penalty before Martin opened the scoring for the Gills, who also knocked the Glovers out of the Capital One Cup on Tuesday, after 35 minutes with a 20-yard free-kick following Nathan Smith's foul on Kedwell.
Smith then brought down Aaron Morris in the box 10 minutes into the second half, and Kedwell finished at the second attempt after Chris Weale parried his penalty.
Yeovil have lost all three of their matches so far this season and boss Gary Johnson said: "We're trying to get them going, but we missed a penalty at a vital time. They had a penalty and a free-kick, and that was it.
"We didn't have enough quality or pace or energy. We looked sapped of a few things. I'm not sure we threatened enough.
"Our quality let us down; not being able to head the ball down from a corner, shoot on target, play a one-two. We got a few decent crosses in at the end, but that was about it.
"It looks like we're going to have to assess this whole situation. We've got to keep building. I know it's only three games - and Gillingham twice - but we're not happy at the moment.
"I've had these situations before and turned it around. It's three games, and that quickly becomes four, five and 10.
"We've got to turn it around. Individuals have got to do a lot of work themselves to show me that they're capable at this level."