Wigan's hopes of reaching the last 32 were hanging by a thread after a costly home defeat by Zulte Waregem two weeks ago that played a major part in Coyle's departure.
That left Rosler's side needing to win in Slovenia and hoping Rubin Kazan, who had already secured top spot in Group D, could beat Zulte.
The Russians did their bit with a 2-0 victory but, having been reduced to 10 men in the first half, Wigan were unable to prevent a sixth consecutive defeat.
Wigan were probing but it took another eight minutes for them to really threaten, Nick Powell striking a volley from outside the box that was dipping but always too high.
Callum McManaman was keen to show his trickery down the right and he whipped in a fine cross that would have provided Powell with a great chance had Soares Arghus not diverted the ball behind.
Wigan continued to increase the pressure and Gomez was extremely unlucky with a free-kick in the 36th minute.
The Spaniard was fouled 25 yards out and picked himself up to curl a shot against the near post.
They were given a huge boost late in the first half, as Manchester United loanee Nick Powell tumbled under the challenge of Maribor defender Arghus, giving Jordi Gomez the chance to squeeze the spot-kick underneath Jasmin Handanovic to keep the visitors' hopes alive.
Almost immediately Maribor responded, but not without controversy.
Damjan Bohar burst forward from midfield before cutting back inside and shooting, with referee Szymon Marciniak adjudging the already booked Chris McCann to have handled the ball, giving him his marching orders with a second booking.
The decision looked harsh on McCann and Wigan, and their sense of injustice was doubled after Scott Carson got a strong hand to Dejan Mezga's penalty, tipping it onto the post before the ball rebounded back out to the midfielder who duly tapped home.
Wigan, who were under the tutelage of former Brentford boss Rosler for the first time, were understandably streched after McCann's dismissal.
Rosler made a change for the second half, sending on Ben Watson, the man who won the FA Cup for the Latics, in place of McManaman.
Not surprisingly, the Slovenians offered a lot more in attack, with Damjan Bohar drawing a diving save from Carson at the near post before Ales Mertelj shot over.
Wigan were trying to hit Maribor on the break and they came close when Jean Beausejour crossed for Powell, but he could only guide his shot wide.
The 10 men eventually succumbed just before the hour mark, when Zeljko Filipovic struck gold with an excellent side-footed effort.
Wigan kept pushing for an equaliser and both Leon Barnett and substitute Marc-Antoine Fortune came close.
Ales Mertelj levelled the numbers up after 73 minutes when he was sent off for two bookings
A draw would have denied Maribor a place in the knockout stages but it was not to be and the home side celebrated their first win over an English side - and a place in the last 32 - exuberantly.
Rosler will now turn his attention to trying to halt Wigan's slide down the Championship when neighbours Bolton visit the DW Stadium on Sunday.
|NK Maribor||Team Statistics||Wigan Athletic|
|1||1st Half Goals||1|
|6||Shots on Target||5|
|8||Shots off Target||6|
|36.1||1st Half Poss.||63.9|
|36.7||2nd Half Poss.||63.3|