The Premier League strugglers, again with Darren Bent on the bench, started well and Belgium striker Benteke signalled his intent with an early header which was easily saved by Wes Foderingham after a good run and cross by Gabriel Agbonlahor.
Villa midfielder Barry Bannan curled a 25-yard drive wide but there was little to choose between the sides in the opening quarter of the game.
It was the home side, in front of a big partisan support, who came close to breaking the deadlock when a Paul Benson shot beat Shay Given in the Villa goal but the Irishman was retrieved by Matt Lowton's goalline clearance.
It proved to be crucial as Villa took control with two goals in the space of 10 minutes before half-time.
Stephen Ireland's left wing cross picked out Benteke and the Belgian, under pressure, headed past Foderingham in the 30th minute.
Agbonlahor doubled the visitors' advantage six minutes before the half-time interval when he accepted another Ireland prompt before speedily drifting in from the left to drill a low shot home.
The England forward then had a decent penalty shout turned down when he tumbled in the box before League One high-flyers Swindon emerged fired up from Paolo Di Canio's half-time talk.
Matt Ritchie was causing havoc on the Villa right and delivered pinpoint centres to Gary Roberts but the winger headed the first chance wide when studiously placed and the post with the second from a yard out.
The introduction of teenager Miles Storey, for James Collins, proved pivotal for the Robins with 15 minutes left as the 18-year-old expertly poached two goals in three minutes.
He was well placed to side foot in a cross (78) and then sensationally backheeled an equaliser two minutes later to send the County Ground wild.
However, it was Paul Lambert's side who booked their place in the quarter-final draw after American full-back Eric Lichaj pulled the ball across goal for Benteke to deliver victory.