Peter Clarke was whistled for a foul on Matej Vydra, and Troy Deeney's spot-kick was the spur for Watford to run riot in the second half.
The result left Huddersfield without a win in 12 Championship games, and Grayson said: "The decision just before half time changed the game. We let it affect us and it obviously gave them a big confidence boost.
"I thought, first half, we were well truly in the game. We kept them quiet, you could sense the atmosphere had gone quiet as well.
"The decision was wrong, it's great tackle by our right back and it's outside the box, so it's a poor decision.
"But in the second half we felt sorry for ourselves and allowed Watford to dictate the game and when you do that against a team like Watford then you're under pressure because they're a good side."
There was more bad news for the Terriers - winger Danny Ward suffered a fractured metatarsal in training ahead of the game and will be out for six to eight weeks.