Kay was left unmarked to head home his fifth goal of the season from a Gary Roberts' corner after 50 minutes. The Terriers were then denied a spectacular winner on 69 minutes when Swindon keeper David Lucas brilliantly tipped over Lee Novak's dipping 30-yard drive.
Huddersfield, who have now drawn four of their last five matches, took the lead on eight minutes through skipper Peter Clarke's freak goal. He beat Simon Ferry to a long throw-in and crossed from the byline, only for the ball to end up in the net from a near-impossible angle.
The Robins were behind for barely two minutes.
On-loan Bolton winger Danny Ward crossed for Charlie Austin to head in unmarked his 12th goal in 15 games since joining from Poole in November.
Ward then caught Huddersfield goalkeeper Alex Smithies off guard with an instant shot on 27 minutes from an acute angle after Billy Paynter forced the home side's Neal Trotman into a mistake.