Alan Knill's League One side, who trailed 3-0 at half-time in the first-round tie, struck twice in time added on to draw level at 5-5 at the end of 90 minutes at Pride Park.
The sides could not add to the 10 goals in extra time but Scunthorpe held their nerve in the shootout.
Richard Keogh, Jake Buxton and Theo Robinson gave Derby a seemingly unassailable three-goal half-time lead before Scunthorpe striker Andy Barcham pulled one back seven minutes after the restart.
Buxton headed home his second within a minute to restore Derby's three-goal cushion, but Scunthorpe hit back through Mike Grella, in the 63rd minute, and Bobby Grant 10 minutes later.
Nathan Tyson headed home Derby's fifth goal in the 83rd minute, but Scunthorpe were not finished and grabbed a fourth through substitute Connor Jennings in the fourth minute of time added on and an equaliser two minutes later when Grant slammed home a penalty following Michael Jacobs' foul on Josh Walker.
Scunthorpe goalkeeper Sam Slocombe kept out Robinson's 72nd-minute penalty after visiting defender David Mirfin's foul on Keogh.
Keogh rose highest to head home Derby's opener in the 30th minute and the home side struck again four minutes later when Buxton headed in Jacobs' free-kick.
Derby appeared to have ended the contest five minutes before the break. Jacobs swung over a corner and Robinson steered the ball home at the near post.
Barcham pulled a goal back for the visitors early in the second half, drilling home a low finish, but Derby restored their three-goal lead within 60 seconds.
Jacobs was again the provider, this time clipping over a free-kick and Buxton headed his second into the top corner.
But Scunthorpe scored their second courtesy of Grella's thumping drive before Grant set pulses racing with a crisp finish from Mark Duffy's cross.
Tyson looked to have ended Scunthorpe's hopes when he headed in a cross from recent signing Paul Coutts with seven minutes left, but Jennings and Grant, from the spot, both struck in stoppage time to ensure another 30 minutes.
When that could not separate the sides Scunthorpe edged it on penalties.
