Terriers midfielder Chris Atkinson gave his side an early lead before goals from Michael Tonge and Luciano Becchio put the visitors ahead.
Adam Clayton levelled from the penalty spot just before half-time but Leeds dominated after the break as David Norris and Becchio pounced to secure victory for Neil Warnock's side.
"We're very disappointed after we started very well. We had a purpose about us and passed the ball well and got a good early goal," said Grayson.
"But we sat back and stopped doing the good things and after their equaliser we were on the back foot.
"We didn't do enough when we had the ball in terms of passing and creating chances and when Leeds had the ball we looked like we weren't going to defend as well as we can do.
"Al round it's a really disappointing afternoon for us. Leeds were sharper than us and won more individual battles."
But Grayson refused to panic and insisted on looking at the bigger picture.
"This is how the Championship is. You can win games and get on a good run, but you have to be careful that you don't get on a downward spiral," he added.
"But a week ago it was all doom and gloom at Leeds and suddenly they win three on the trot and everything's rosy, so we won't get carried away when we're in the top six and we won't get too down now."