Celtic dominated from the off and their pressure was rewarded when the Finnish striker drove the ball low between the legs of defender Frazer Wright and keeper Alan Mannus in the 11th minute.
Adam Matthews stung the keeper's hands with a fierce drive in the 17th minute and Derk Boerrigter missed a golden chance to double the advantage three minutes later.
But the home fans were cheering again soon after when Charlie Mulgrew gave the Northern Ireland international no chance with a low drive from 25 yards in the 26th minute.
Celtic maintained their stranglehold on the game and Anthony Stokes and Boerrigter were both inches away from adding a third before half-time.
After the break Stokes strolled past two defenders and fired narrowly wide with Mannus well beaten as Celtic picked up where they left off.
But Saints came more into the game as the half wore on and in-form Stevie May headed over from close range after good work from David Wotherspoon on 67 minutes.
And they were rewarded for their pressure nine minutes from time when Liam Caddis held off the attention of Efe Ambrose to bundle home at the back post.
The nerves felt around the ground did nothing for the players' composure and the earlier domination was replaced by hesitancy which almost allowed Saints to leave Parkhead with a point.
With 90 minutes on the clock, May rattled the home post and Dave Mackay saw his follow-up saved superbly by Fraser Forster, who ensured the Hoops moved joint top after Inverness gave up their unbeaten record at Aberdeen.