Chris Wood was twice denied by the frame of the goal in a five-minute spell either side of half-time while Steven Caldwell also saw a first-half header strike the post as Birmingham came closest to breaking the deadlock at the Estadio da Madeira.
The npower Championship side, competing in a major European competition for the first time since 1962, gave a debut in midfield to new signing Jonathan Spector while 17-year-old Nathan Redmond had his first start of the season on the wing.
Portuguese hosts Nacional started the better of the two sides and a great challenge by Curtis Davies early on denied Mateus da Costa what might have been the opening goal.
However, Birmingham eased their way back into the contest and they could consider themselves unlucky not to have held a half-time advantage.
Davies brought a save out of home goalkeeper Elisson before Caldwell and Wood both struck the goalframe.
Caldwell was the first man to be denied in the 24th minute when his far-post header from Jean Beausejour's corner hit the inside of the post.
Wood was also left cursing his luck a minute before the break when he hit the crossbar following a Beausejour free-kick.
The visitors carried that momentum into the second half and, inside two minutes of the restart, they hit the post yet again.
Redmond's drive from outside the box was parried by Elisson but Chris Burke picked up possession and his cross was headed onto the upright by Wood.
Wood then fired an effort across the face of goal before Nacional tried their luck with efforts from Mateus and Diego Barcelos, although neither troubled Boaz Myhill.
Birmingham came close to snatching a late winner when Caldwell got on the end of Beausejour's free-kick but his header went over as the game finished goalless.