The on-loan Coventry midfielder marked his first Scottish Premiership start for the Saints with a first-time effort in the 54th minute to hand the visitors a 1-0 victory.
Swanson showed brilliant technique to fire the ball into the top corner from 25 yards after Virgil van Dijk had headed David Wotherspoon’s corner out of the box.
The goal sent St Johnstone into the top six and gave Aberdeen fresh hope in the title race following their 4-0 defeat at Parkhead on Sunday.
Celtic had the chance to move nine points clear but they remain two victories ahead of the Dons with their games in hand now used up.
The Hoops never hit the heights that had earned them 10 domestic victories in a row this year and Saints goalkeeper Alan Mannus was well worthy of his clean sheet after making some important saves.
Celtic, who were without the injured Kris Commons and John Guidetti, started with purpose and Stuart Armstrong had a shot blocked and the recalled Anthony Stokes curled wide from a tight angle following a one-two with Leigh Griffiths.
The chances continued to come and Saints soon got a sight of goal when the Celtic central defence failed to deal with a long ball, but Steven MacLean slice wide from 25 yards.
Mannus tipped over Gary Mackay-Steven’s first-time curler at the other end and Griffiths fired just wide from 30 yards.
St Johnstone’s pressure ultimately led to the corner from which Swanson stunned Celtic Park in the 54th minute.
Celtic tried to respond quickly and Stokes earned himself a yard of space on the edge of the box, but Mannus got down to hold his drive.
And Steven Anderson had a great chance to double St Johnstone’s lead moments later when David Wotherspoon’s deep corner found him unmarked but the centre-back volleyed wide of the near post.
Mannus produced some fine saves late on, parrying Ambrose’s header before getting down to stop Bitton’s drive.
Van Dijk headed just wide from a corner in the dying minutes as the Saints clung on to hand Celtic their first defeat of 2015.