Baxter's fine free-kick sealed their place in the hat after Cristian Montano's early second-half strike had finally ended Harriers' resistance.
Oldham controlled the first-half but when they found the net after 26 minutes, Matt Derbyshire flicking home Lee Croft's near post cross, it was ruled out for offside.
Before that, Baxter, at the heart of most of Oldham's best moves, had seen three chances to put the Latics ahead come and go. Twice he was foiled by Harriers keeper Nathan Vaughan, while a 15-yard volley flew over.
Kidderminster's best moments came either side of half-time as Keith Briggs curled just wide from 25 yards and Anthony Malbon headed a sitter straightforward chance at Dean Bouzanis from six yards.
Kidderminster were soon made to pay for that miss as Montano tucked home after Derbyshire slipped him through on goal. And Baxter sealed Oldham's progress into the second round, his clever free kick from out wide catching out Vaughan at his near post.
