The Robins shocked their hosts in the opening period as Collins pounced upon a mistake by Potters goalkeeper Thomas Sorensen to make it 1-0 in the 27th minute and then doubled their lead by nodding in just prior to half-time.
Stoke salvaged extra-time after the break through strikes from Kenwyne Jones and Jon Walters, but Paulo Di Canio’s Swindon would not be denied in the additional period, with defender Aden Flint (105) and Collins (119) netting either side of Peter Crouch’s header (111).
Stoke were playing their second match in just over 48 hours following Sunday’s 0-0 draw with Arsenal and manager Tony Pulis refreshed his first XI by making six changes from that game.
The Potters looked anything but invigorated in the first half, though, against a lively Swindon team whose record for the campaign going into the contest showed no defeats and no goals conceded in four matches, and included a 3-0 victory over Championship outfit Brighton in the previous round.
The Robins appeared comfortable from the off and after Tommy Miller had sent a shot into the arms of Sorensen, Rafaelle De Vita struck wide.
It seemed only a matter of time before the hosts settled down and they issued a couple of warnings at the other end, Cameron Jerome firing in a speculative effort from distance, Jones heading one cross over and the pair then getting in each other’s way as they attempted to meet another.
Stoke were showing some sloppiness at the back, though, with Collins just unable to capitalise on one defensive lapse before he made the most of a further error to put Swindon ahead.
Jay McEveley delivered the ball towards the box where it was fumbled by Sorensen, allowing Collins to guide it into the net.
The Potters needed to wake up but it was the visitors who continued to dictate proceedings, with Andy Williams drilling at Sorensen from the edge of the area.
Anything Stoke were creating Jones was heading too high and four minutes before the break, they were a further goal behind.
Collins was the scorer once again, nodding in De Vita’s corner, and the Republic of Ireland Under-21 striker nearly completed his hat-trick moments later, seeing his shot cleared off the line.
The attacking flurry from Swindon did not let up and De Vita hit the post just before the whistle went for the interval, met by boos from the Stoke fans.
Pulis opted not to make any substitutions at half-time and his team continued to look shaky in defence as the second period got under way, although there was a decent block made to deny De Vita.
Going forward, wide men Jermaine Pennant and Michael Kightly were doing their best to apply some pressure and in the 63rd minute, the latter provided the cross as Jones demonstrated some accuracy to head Stoke back into the game.
Pulis made a treble substitution in the 77th minute with Crouch, Walters and debutant Jamie Ness coming on for Jones, Jerome and Ryan Shotton and the reshuffle soon paid off, Walters stroking the ball in with four minutes of normal time remaining.
Stoke finished the 90 minutes with plenty of momentum, but Crouch and Pennant were unable to apply the finishing touch and the game went into extra-time.
The Potters continued to look threatening into the additional period, but Swindon sprang a shock on the stroke of half-time as Flint headed in at the back-post to put them in front once more.