Only some brave last-ditch defending stopped them going behind but when Gareth McAuley rose to head home Chris Brunt's cross in the 40th minute, a famous victory looked on the cards.
But Serbia, playing in the near-deserted Crvena Zvezda Stadium as a result of crowd trouble against Italy, hit back through Marko Pantelic before Zoran Tosic drilled in the winner via a slight deflection.
With just four minutes gone Brunt attempted something spectacular, lashing an ambitious looping shot at goal from just inside the Serbia half.
It was directed well enough to cause goalkeeper Zeljko Brkic a moment's hesitation but it continued harmlessly over.
Brkic twice flapped at set-pieces in the space of a minute, first Sammy Clingan's curling free-kick then Brunt's corner.
The West Brom man next tried his luck from 20 yards but, after showing good control, his low drive flew wide.
Serbia almost took the lead after 15 minutes, McAuley's last-gasp slide steering what looked a certain goal over the bar.
Milos Krasic and Dejan Stankovic started the move on the right wing before the ball was cut back for Milan Bisevac, whose goalbound effort was brilliantly diverted by McAuley.
Northern Ireland somehow managed to get out of jail again after 22 minutes, Tosic jinking into the box before teeing up Adem Ljajic.
With the goal again gaping, a combination of Aaron Hughes and McAuley smothered the danger.
Hughes was in some discomfort having gone down with an apparent shoulder injury but he remained on the pitch after treatment.
Krasic was emerging as Serbia's key man, constantly asking questions of the defence, but the odd loose pass hinted at frustration in the home ranks.
Chelsea's Branislav Ivanovic attempted a cheeky chip after 33 minutes, but debutant goalkeeper Lee Camp was grateful to watch the ball creep wide.
Jonny Evans continued his side's rearguard with a vital tackle as Tosic looked ready to pull the trigger inside the penalty area.
An innocuous foul earned Brunt the chance to deliver a free-kick from the right flank five minutes before the interval and he found McAuley eight yards from goal.
The Ipswich defender timed his jump well and aimed for the top corner, beating Brkic to send the 265 travelling fans into raptures.
Serbia, with no support of their own following the UEFA ruling, were shellshocked and almost conceded again when Brunt played in Kyle Lafferty.
The Rangers forward took a neat touch to create some room but stabbed his shot over the crossbar as Brkic advanced.
Lafferty did not emerge for the second half, replaced by Northern Ireland's record scorer David Healy.
Serbia boss Vladimir Petrovic also went to his bench during the interval, with out-of-favour Liverpool forward Milan Jovanovic and Milos Ninkovic entering the fray.
Northern Ireland were close to a second in the 49th minute, Healy and Brunt linking well before Johnny Gorman whipped in a cross from the left. Brunt was waiting in the centre but Neven Subotic deflected the ball wide of the post.
Camp ensured the lead remained at 1-0 when Jovanovic met a Krasic corner, heading powerfully at goal. The former England Under-21 international's reactions needed to be good and he did not disappoint, standing firm to beat the ball away.
Healy, occasionally a little off the pace, picked up a needless yellow card to rule himself out of Tuesday's clash with Slovenia.
Worse was to follow in the 65th minute when Pantelic swept home from eight yards after a flowing move involving Ninkovic and Jovanovic.
The Liverpool man almost put the hosts ahead seconds later after finding space in the penalty box but he hesitated to shoot and the chance fell away.
Northern Ireland were now looking in trouble every time Serbia attacked, their swagger finally returning.
The second goal came after 73 minutes, Stankovic charging towards the danger area before laying the ball to Tosic.
The CSKA Moscow midfielder took a touch before rifling his shot into the near corner via a slight deflection off Craig Cathcart.
Having taken the lead Serbia eased off a little but Northern Ireland could not take advantage as substitutes Paddy McCourt and Warren Feeney struggled to make an impression.