Former Swansea and Portsmouth full-back Duffy scored unmarked at the near post from a Ryan Harley corner after 37 minutes.
A corner whipped in by Harley five minutes later nearly put the Grecians two up, but Troy Archibald-Henville's header was ruled out by referee Nigel Miller for a foul on Peter Clarke.
The Terriers were smarting before kick-off after multi-millionaire businessman Hoyle effectively accused the team of being cowards in his match programme notes.
Hoyle, away in New York on business this weekend, feels the team wilt in the face of physical opposition and he is unlikely to be impressed when he hears about their latest lacklustre display.
A Jordan Rhodes header off the bar from a Graham Carey cross was about as good as it got for Town in the first half. Rhodes, Scott Arfield and Anthony Pilkington had chances after the break while Arfield struck the bar in stoppage time with a 20-yard belter as Town finally woke up.