Will Buckley scored both goals for Gus Poyet's side and despite Simon Church pulling a goal back in the final minute of the contest, Brighton held on to claim all three points.
"I was disappointed at how we started the game. We know they like to start really positive and we had to close them down, but we were two or three yards off," he said.
"The first 20 minutes probably lost us the game because we allowed them to get into a pattern of play.
"We had a few chances through Simon Church and Lee Novak and we said to the players at half-time that if we got the next goal we had a chance to get something.
"We did get the goal but it was too late to salvage something."
Grayson was also quick to praise Brighton's performance, as they moved above Huddersfield in the Championship table.
"You always know Gus' teams are going to be composed and we probably contributed to our own downfall by not getting at them," he added.
"It's a learning curve for us and I have told the players that when we lose matches, we have to understand why we've lost so that we can get better.
"Brighton have spent some money over the last couple of years with players by (Craig) Mackail-Smith and Buckley, and if they are not thereabouts at the end of the season they will be disappointed."