Cheltenham led 1-0 at half-time with Lloyd Owusu heading the Robins ahead on his debut but the Terriers hit back with goals from Nathan Clarke and Ian Craney to lift themselves off the bottom of the table.
Cheltenham were the better side in the first half and went in front when Alex Russell sent over a left-wing cross and Owusu powered his header home from eight yards.
But Huddersfield were the better side throughout the second half and deservedly drew level after 57 minutes, although they were helped by a controversial refereeing decision.
David Unsworth fired in a volley which clipped the bar, but the referee gave a corner, from which Clarke stabbed home the equaliser.
The winner came 10 minutes later when Cheltenham failed to deal with another corner, the ball falling to Craney 25-yards out and he drilled a superb shot past Shane Higgs.