After a goalless first period at Pilot Field, the hosts were awarded a spot kick in the 47th minute when Matt Bloomer was adjudged to have fouled Zac Attwood in the penalty area.
Lee Carey powered home from the spot to give his side the lead and Hastings looked like they were going to claim a narrow victory before the visitors restored parity in the final minute.
Tom Platt latched onto a pass from Dave Merris before firing into the bottom-right corner to send the game into extra-time.
And with extra-time producing no further goals, the match went to penalties with Harrogate converting the opening spot kick.
Having netted the dramatic, late equaliser, Platt was unable to find the target from the spot and unfortunately for him, his miss turned out to be the turning point as all the following penalties were converted successfully.
Hastings will face Middlesbrough in the third round.