Mathias Janisch stabbed the ball home just three minutes from time in an exciting encounter in the Grand Duchy.
The home side had earlier led for five minutes when Gareth McAuley made it 2-2 with eight minutes to go, only for Janisch to pop up at a free kick and grab their first World Cup win on home soil in 43 years.
Martin Paterson gave Northern Ireland an early lead which was cancelled out on the stroke of half time by Aurelien Joachim.
Stefano Bensi put Luxembourg ahead on 77 minutes, then McAuley equalised before Janisch's dramatic winner.
Prior to kick-off in the Josy Barthel Stadium, O'Neill's side could look back on some of the brighter signs from their recent performances.
They were leading Portugal 2-1 in Belfast until a Cristiano Ronaldo hat-trick and the indiscipline of Chris Brunt and Kyle Lafferty blew away hopes of a shock win. That had been preceded by Northern Ireland's only win of the qualifying campaign, at home to Russia, but their lacklustre draw with perennial strugglers Luxembourg on home soil a year ago typified a disappointing attempt to make it to Brazil in 2014.
The hosts started brightly following a pre-match downpour that had left a slick playing surface. The Northern Ireland defence allowed Lars Krogh Gerson to head powerfully wide from a corner, and seconds later Daniel Da Mota wriggled into space in the area but his shot failed to hit the target.
But Paterson calmed his side's nerves with the opening goal, the Huddersfield striker latching on to Shane Ferguson's well-placed pass from midfield before wrong-footing two defenders and then curling his shot past home goalkeeper Jonathan Joubert.
Luxembourg's main threat came from long range shots and Northern Ireland keeper Roy Carroll could only watch as, on the half hour mark, Gerson curled a cute shot over him. Thankfully for the 1,200 travelling fans, his effort only rattled the bar.
Luxembourg continued to create more chances, and Carroll was forced to parry away a rasping drive from the dangerous Bensi as the home side strived to bely their FIFA ranking of 140.
But with referee Robert Malek about to blow for half time, David Turpel's pass allowed Joachim to home in on goal before his shot looped over Carroll having deflected off the sliding Lafferty. It was what the spirited hosts deserved and capped a good fightback following Paterson's opener.
Bensi drove a free kick just wide of the post early in the second half, and as the game developed an end-to-end pattern, Jamie Ward saw his penalty claims waved away.
Substitute Billy McKay found the side-netting from close range just a few minutes after replacing Ferguson and Northern Ireland paid the price when the home side took the lead with 12 minutes left.
The impressive Bensi was allowed enough time and space to shoot from outside the area, and Carroll could not get a hand on the ball as it bounced past him to give Luxembourg the lead.
McAuley looked to have spared Northern Ireland's blushes five minutes later when his powerful downward header from a free kick bounced past Joubert.
However Joachim climbed well to knock down a free kick, and Janisch pounced with three minutes of normal time remaining to make a miserable qualifying campaign for Northern Ireland even worse with two testing away trips still to come.