Aurelien Joachim's goal in the 87th minute gave the perennial international strugglers a win in a World Cup match for only the fourth time in their history.
The home side had overturned Martin Paterson's early goal, but despite Gareth McAuley's equaliser eight minutes from time they hit back to claim a famous victory.
"It was very poor, we were disappointing from start to finish," O'Neill told Sky Sports.
"We were fortunate to go ahead and we didn't learn from that, we lost a poor goal before half-time. Then when we get ourselves level, we lose another from a set-piece.
"The performance was unacceptable, the energy and work rate was not at the level required."
A win over group leaders Russia in August and a spirited display in defeat to Portugal last Friday had raised hopes that Northern Ireland could make advances in their final three qualifying matches.
However, their manager felt the progress made recently had been thrown away by his side's poor performance in the tiny Grand Duchy.
"This is a team that needs to find ways to win games," O'Neill added.
"We've had a lot of plaudits in the last couple of games but the reality is it counts for little when you come and play like tonight.
"It's okay lifting it for Portugal and Russia but we needed that tonight. When we're down gears we're average.
"In these games when we're expected to win, we need to bring the basic ingredients to win a football match.
"We didn't manage the game well when we were on top and we didn't deserve anything from the game. Luxembourg were the better team on the night."