Town looked poised to pick up three points following their disappointing home draw against Bury on the opening day but Grimes' late finish left them deflated come the final whistle.
The visitors had completely bossed the first half and capitalised on the hosts' habit of conceding early goals when Lee Novak linked up well with Scott Arfield before rifling a low drive past Jake Kean.
Both Arfield and Novak had shots cleared off the line as the Terriers pushed for a second, but the hosts levelled just before the break courtesy of Jean-Louis Akpa Akpro's volley.
Kean kept out a stinging shot from Gary Roberts early in the second period but the hosts fell behind again after 69 minutes when Novak drilled a low cross into the six-yard box and substitute Danny Ward stole in ahead of Joe Widdowson to clip the ball into the net.
Kean pushed a long-range effort from Oscar Gobern over the bar as Town tried to wrap up the result - but Grimes' late strike ensured more frustration for Terriers boss Lee Clark.