Tommy Rowe gave Posh a first-half lead but two goals in four minutes after the break from Michael Gash and Jack Byrne turned the game on its head.
A Britt Assombalonga penalty levelled the scores with 16 minutes to go but parity lasted just two minutes as Joe Lolley grabbed a third for Harriers, who are now the only non-league side left in the competition.
In the all-Premier League clash at Craven Cottage, Fulham secured their progression with a 3-0 win over Norwich.
Darren Bent netted the opener for the Cottagers and Ashkan Dejagah doubled the lead four minutes before the break.
Steve Sidwell then added a third in the second half as Fulham earned themselves a trip to Sheffield United in round four.
Conference outfit Macclesfield had their cup run ended as three late goals saw them beaten 4-1 at Hillsborough by Sheffield Wednesday.
Chris Maguire delivered an early blow to the non-league team's hopes as he fired the Owls ahead after just four minutes, but a 65th-minute penalty from Scott Boden had the Silkmen level.
Jacques Maghoma settled the nerves at Hillsborough, though, as he netted with 12 minutes to go and Jermaine Johnson and Miguel Llera sealed the victory late on.
A Joe Garner hat-trick handed Preston a shock 3-2 win over Championship side Ipswich at Deepdale.
Franck Nouble put Ipswich into the lead after 58 minutes but two goals in as many minutes from Garner put Preston in control before substitute David McGoldrick grabbed an equaliser.
Garner then completed his hat-trick with two minutes remaining and Preston will now go on to face Nottingham Forest.
Watford had no such problems overcoming Bristol City as goals from Marco Davide Faraoni and Lewis McGugan gave them a 2-0 win.
Port Vale will face Brighton in round four after they claimed a dramatic 3-2 win over Plymouth.
Argyle were ahead after just two minutes through Andres Gurrieri but Jordan Hugill equalised before Conor Hourihane restored the lead nine minutes before the break.
Ben Williamson then levelled things up again and the game took another twist as Jennison Myrie-Williams fired Vale into the next round before Chris Lines was sent off.
Championship teams Birmingham and Bournemouth both made serene progress into round four with comfortable wins.
Brett Pitman gave the Cherries the lead five minutes in against Burton but Jimmy Phillips had Albion level before Tommy Elphick restored the lead on the stroke of half-time.
Ryan Fraser opened up a two-goal cushion in the 86th minute and two minutes later a Pitman penalty secured a 4-1 win and a fourth-round tie against Liverpool.
At St Andrews, Paul Robinson scored for the Blues after 35 minutes and two quick late goals from Chris Burke sealed a comfortable 3-0 triumph over Bristol Rovers.
Charlton and Oxford will have to play again after their re-arranged third-round tie ended 2-2.
The first game had been called off as the Addicks continue to have problems with the pitch at the Valley and their woes were deepened when Johnny Mullins fired United ahead after 13 minutes and Scott Davies doubled the lead 11 minutes later.
Michael Morrison pulled one back in the 54th minute to give the home side hope, though, and they were level in the 82nd minute through Yann Kermorgant.
Holders Wigan needed extra-time to complete a 3-1 replay win against MK Dons. The two-sides had shared a dramatic 3-3 draw at the DW Stadium but it looked for a long period like one goal would win it at Stadium MK after Luke Chadwick fired the Dons ahead after 10 minutes.
Manchester United loanee Nick Powell grabbed an equaliser with 11 minutes left to send the tie to extra-time, though, and then netted again with Marc-Antoine Fortune grabbing the third.