Club Brugge and visiting Braga went through to the last 32 after their 1-1 draw in Belgium, finishing one point ahead of Chris Hughton's side.
It was a brave effort by Blues, for whom Rooney settled the encounter with a first-half header - his first goal for four months - from Nathan Redmond's fine cross.
Redmond caused all kinds of problems throughout the 90 minutes on the right flank and the 17-year-old supplied a string of teasing balls into the Maribor box.
On this evidence he will have a big part to play when City resume their challenge for promotion from the Championship.
Only the woodwork denied him a deserved goal four minutes from time with a stunning 25-yard volley.
Birmingham, who made nine changes from the side in action against Doncaster at the weekend, were the first to settle and Redmond quickly made his presence felt.
The winger delivered a deep cross in the direction of Nikola Zigic and it needed Maribor defender Zeljko Filipovic to make a vital interception.
Redmond again got to the byline and asked questions with his centre but keeper Jasmin Handanovic was up to the task.
The teenager skipped past two challenges and cut inside but this time his centre was sliced and Nejc Potokar was able to side-foot to safety.
Morgaro Gomis was brought down by Ales Mertelj 25 yards out and David Murphy's free-kick deflected off the defensive wall and looped over the bar.
Jovan Vidovic headed just wide from a corner in a rare Maribor raid and then at the other end made a crucial tackle to deny Zigic from a Jean Beausejour cross to the near post.
After 24 minutes Adam Rooney deservedly put Blues ahead.
Redmond was the supplier with another testing ball, skipping past Arghus, and this time Rooney headed home from close range.
Rooney, whose goal was his first since August, then got on the end of a deep Beausejour cross at full stretch.
His shot from a narrow angle deflected off Handanovic for a corner.
Zigic looked short of confidence and lacked conviction when shooting across the face of goal after Aleksander Rajcevic backed off him and allowed him space.
Birmingham began the second half on top and a Redmond cross was knocked down by Zigic, though no-one was able to take advantage.
Ibanez needed to make a timely interception to flick the ball away from Dejan Mezga inside the City box.
Redmond continued to look impressive and tested Handanovic with a long-range drive.
Murphy had a shot blocked by Rajcevic and Davies failed to make contact with his header with the goal at his mercy from a Redmond corner.
Zigic had a header deflected over the bar after more good wing play by Redmond, who went on to find Rooney with a corner only for Handanovic to make another parry.
The woodwork denied Redmond a deserved goal when his stunning 25-yard volley crashed against the post.
He left the field to a standing ovation when replaced by Chris Burke with two minutes remaining.
