The Scotland international scored the first goal of extra time before Don Cowie, who had headed a late equaliser from a Conway cross to send the match into an extra period, sealed victory.
Conway had set the tone for a high-scoring contest when he was provider for Cardiff's opening two goals, scored in as many first-half minutes by Gabor Gyepes and Jon Parkin, to give them a 2-0 interval lead.
Huddersfield hit back through Jordan Rhodes and Daniel Ward before the former appeared to have settled the tie with two minutes to play.
But Cowie struck four minutes into injury time to force the extra half-hour - six minutes into which Conway drove at the Terriers' defence and unleashed a 25-yard effort that beat visiting goalkeeper Ian Bennett with the aid of a deflection.
Cowie made sure sure of the win in the 117th minute by thrashing home after Bennett and Gary Naysmith had thwarted efforts from substitute Kenny Miller.
Bluebirds boss Malky MacKay rung the changes for Cardiff, with only Cowie and Conway remaining from the side that drew with Burnley, while Ward replaced Gary Roberts in the only change to Huddersfield's weekend line-up.
The home side led in the 16th minute when some slack Huddersfield marking allowed Gyepes to head home from Conway's right-wing free-kick.
Parkin drew the foul from Peter Clarke that made the opener possible and he doubled Cardiff's advantage moments later.
Conway was the provider once more, this time from the left flank, as Parkin planted a header beyond Bennett via the upright.
Huddersfield's woes were compounded when Lee Novak limped off to be replaced by Alan Lee in the 26th minute. But they went about turning the contest on its head eight minutes after the break.
Rhodes charged down a clearance from Bluebirds defender Paul Quinn, collected the loose ball and finished from an acute angle.
Huddersfield levelled when Ward drove into the bottom corner from the edge of the area after efforts from himself and Oscar Gobern were blocked.
And Rhodes thought he had completed a remarkable comeback for Town as Gobern sent him clear and he slotted past Heaton.
But in the final minute of stoppage time Cowie became the third Cardiff player to head home from a Conway cross.
The former Dundee United winger then got the goal his display merited and before Miller struck the woodwork and Cowie added close to the scoreline.