Novak struck the winning blow in the 93rd minute with a brilliant volley after Kay headed in a Gary Roberts corner to tie the scores at 3-3.
There was even time for Terriers striker Theo Robinson, who put Town ahead with a first-half penalty, to hit the bar with a lob in the 97th minute.
Walsall's Mark Hughes was sent off for a foul on Jordan Rhodes which led to Robinson's penalty after 21 minutes and Rhodes doubled the advantage just before the break with a cool finish from Danny Drinkwater's long ball over the top.
But Walsall responded with two goals in four second-half minutes from Troy Deeney and an exquisite volley from substitute Alex Nicholls seven minutes from time.
In a exciting contest packed full of incident, Walsall striker Darren Byfield had a first-half effort ruled out for a handball after Emmanuele Smith's header was parried by Huddersfield keeper Alex Smithies.