The Cherries sat deep to soak up the hosts’ early pressure at Priestfield but they were undone when Curtis Weston drilled Gills in front from 30 yards after Joe Martin’s 21st-minute corner was only half-cleared.
But the visitors became more adventurous in the first-round replay, and could have been level at half-time had Gills keeper Ross Flitney not blocked Stephen Purches’ six-yard effort.
With the Cherries piling forward, the equaliser finally arrived after 55 minutes when Andy Frampton steered Scott Malone’s cross into his own net.
Flitney denied Marc Pugh from point-blank range to keep Gillingham level before Charlie Lee found Garry Richards to bundle home after 75 minutes.
Payne all but confirmed a date with Orient eight minutes from time by lashing past Darryl Flahavan but Harry Arter gave Bournemouth a lifeline with a 90th-minute free-kick.