The non-league hosts had led with just five minutes to go but Martin Paterson levelled for the Terriers before Aswad Thomas put through his own net to spare Huddersfield's blushes.
"I walked into the dressing room after the game and the players were genuinely disappointed, and rightly so," said Hurst.
"It is very hard to take and that's just a measure of the performance that our players put in. We more than deserved at least a replay, but that's what can happen.
"The game was wide open and we never settled for anything. We were pushing and at 2-1 we had a couple of chances to make it 3-1.
"I still wouldn't have been comfortable on the sidelines, but obviously a two-goal margin would have given us a little more breathing space.
"But I couldn't be prouder of the players - they put in an exceptional shift and caused Huddersfield a lot of problems after a very difficult start for us.
"From their very first attack, James McKeown has had to make a good save and it was probably 15 minutes before we had our first passing move."
The hosts, looking to make the fourth round for the first time since 1998, took the lead when Ross Hannah tapped home from six yards on 25 minutes.
Oliver Norwood equalised for the Championship outfit, though, with a stunning free-kick six minutes after the break.
But on 62 minutes Grimsby captain Craig Disley headed home from 10 yards to restore their lead.
However, Paterson, who spent three months on loan at Blundell Park over seven years ago, side-footed home and Thomas diverted Huddersfield substitute Duane Holmes' left-wing cross into his own net in the final minute.
"We started off well and I thought we controlled the game for long spells in the first half," said Huddersfield boss Mark Robins.
"But we knew they were going to put us under pressure - it's the FA Cup, and that was a real FA Cup tie and like a throwback to when I was playing.
"The pitch was poor and they were tough conditions - it was all the things that you want as a neutral in an FA Cup tie.
"But as the manager of an away team coming to a Conference side, it's not what you want at all.
"We knew it was going to be tough, they put us under a lot of pressure but had we scored early on with one of the numerous chances we had we might have had a lot easier afternoon.
"There were lots of cameras around looking for an upset, and we disappointed them, which is great."