The Championship side had to come from behind after summer signing Lionel Ainsworth made an immediate impact from the bench, putting Rotherham ahead in the 52nd minute with an unstoppable volley from the edge of the box.
Tigers boss Steve Bruce's own super sub Aaron Mclean spared his blushes with an equaliser after 70 minutes.
Cameron Stewart, who was superb throughout, got to the byline and cut the ball back for Mclean to finish from seven yards out.
Rotherham goalkeeper Scott Shearer kept his side in it with a handful of fine saves, most notably to tip Stewart's shot from an acute angle on to the crossbar in the first half and to tip a Mclean header over in extra-time.
He also denied Matty Fryatt on a couple of occasions.
Stewart should have settled it inside 90 minutes, going through one-on-one late on but his shot beat Shearer and also went past the post.
Big summer signing Nick Proschwitz and Robert Koren produced tired-looking finishes in extra-time, both missing the target.
Mark Bradley missed in the shoot-out with the 14th spot-kick, his effort coming back off the crossbar as Hull emerged victorious.
