Saints went into the second leg of their third qualifying round protecting a 1-0 lead following last week's win in Belarus, but they squandered several chances to put the tie beyond reach.
Milos Rnic's volley in the 75th minute forced extra time, and after a goalless added half-hour, Dave Mackay, Paddy Cregg and MacLean also missed from the spot.
Alan Mannus had kept the Scots in the shoot-out with saves from Vital Kibuk and Siarhei Kazeka, but MacLean's attempt was beaten away by Vladzimir Bushma and the visitors claimed a 3-2 victory.
"I tried to go for power but the keeper has guessed right," said MacLean. "That's football. It's easy when everything is going well.
"But this is what it's all about now. When things go wrong, it's the real players who come out fighting. We've got guys in there who will come out fighting and I'm sure we will.
"We're all gutted in there. They were a stuffy team to play against. They just sat with everybody behind the ball. They didn't show any ambition.
"To be fair they got their goal but I think it was their only shot on goal. That's football - it kicks you in the backside sometimes. It's done that to us tonight.
"They kept fouling us just as we were going through and it was one of those nights. I think they should have had a man sent off.
"Early on the boy fouled me when I was clean through. The referee tried to say there was four defenders. But I don't know where he got that from. We've been unfortunate, to be honest.
"I don't know what happened with the one at the end but the boys have said it was a stone-waller. The referee didn't give us much."