The visitors trailed to a brilliant Wilfried Zaha strike, despite having a man advantage after Palace's Damien Delaney was sent off with 11 minutes gone.
The Terriers also missed a penalty before Keith Southern levelled, while Anthony Gerrard received his marching orders late on.
Grayson said: "You're happy to get a point when you come down to Crystal Palace, a team which are flying at this moment in time.
"But when you look back and reflect on the game, we had some good opportunities. Their keeper made some fantastic saves and we had an opportunity to score from the penalty.
"Anthony Gerrard's sending off? Not from where I am and not from what I've seen on the TV. He didn't jump in, he didn't lead with two feet so I think the referee's got that decision wrong.
"We're disappointed with the decision which got Anthony sent off.
"I would have thought we will appeal but you never know. We had one of those at Charlton and we thought we'd get Keith Southern off but we didn't. You don't hold too much breath."