The visitors took a 54th-minute lead through James Gray in what was their fourth Group 9 fixture but Serbia then netted thrice in quick succession thanks to David Morgan's own-goal (63) and efforts from Luka Milunovic (64) and Uros Vitas (74).
The home side were on the attack from the early stages, with Milunovic passing up a great opportunity by missing the target in the third minute before Milos Jojic brought a save out of Conor Brennan in the sixth.
Northern Ireland's first attempt at goal came in the 10th minute, with Ryan Brobbel trying his luck unsuccessfully, and Serbia were then swiftly back on the front foot.
Following a flurry of efforts at the visitors' goal, Serbia went the closest yet in the 23rd minute when Aleksandar Mitrovic hit the woodwork.
Aleksandar Kovacevic then had a shot blocked and the hosts' pressure continued with efforts from Nikola Stoiljkovic and Jojic keeping Brennan busy.
Northern Ireland were struggling to muster much at the other end, but Brobbel did miss with a free-kick, Gray and Kirk Millar had efforts blocked and towards the end of the first half Carl Winchester produced an unsuccessful attempt.
They began in lively fashion after the break as, either side of Stoiljkovic testing Brennan, Serbian players got in the way of shots from both Gray and Millar, and they were soon in front as the former finished from Luke McCullough's assist.
Serbia quickly looked to bounce back as Brennan kept Milunovic at bay, and within moments of Morgan having a free-kick saved, things were level with the Nottingham Forest man having put the ball into his own net.
Seconds later the game had been fully turned on its head, with Mitrovic setting up Milunovic.
Efforts from Kovacevic and Mitrovic followed and after Morgan failed to hit the target with a chance, Vitas showed better accuracy to add Serbia's third.
Northern Ireland remained under pressure thereafter, with Mitrovic failing to convert a particularly decent opportunity, while Jonny Gorman's blocked strike was the only response.