Saints will meet Dundee United - who beat Rangers 3-1 on Saturday - in the final.
Aberdeen took the lead in the first half after a superb move, Niall McGinn coolly slotting home after 16 minutes.
But May equalised on 61 minutes and then scored the historic winner with six minutes left to book their place in the final at Celtic Park on 17 May.
The victory ends St Johnstone's seven-match losing streak in semi-finals.
After a scrappy, nervous opening to the game at Ibrox, the first moment of quality produced the first goal.
Peter Pawlett, the Scotland Under-21 playmaker, controlled Russell Anderson's pass out of defence on the edge of the Saints box before spinning to pick out McGinn's run through the middle of the opposition defence.
McGinn latched on to his perfectly weighted pass and poked the ball past keeper Alan Mannus.
The Dons should have had a second shortly after. Steven Anderson went to ground far too easily just as McGinn slipped by him before slotting in Adam Rooney, but Mannus was quick off his line to block.
May had his first sight of goal on 23, but saw his shot deflected away from goal by Jamie Langfield's outstretched foot.
David Wotherspoon fired a free-kick on target but the Dons keeper saved with ease, while a May set-piece four minutes before half-time was just as easily gathered.
McGinn should have made it 2-0 after half-time but he miskicked horribly after Pawlett's superb cut-back.
And the Dons youngster would soon regret that mistake as May put St Johnstone back on level terms after 61 minutes.
Aberdeen conceded a needless corner and despite Barry Robson clearing the first delivery, James Dunne returned the ball into the box for May who turned one way, then the other before smashing a low shot past Langfield from an improbable angle.
Robson had a chance to snatch the lead back shortly after but his header, from Willo Flood's cross, was too weak and Mannus saved.
McGinn then headed wide at the back post as Robson sought to make amends with a fine delivery from wide on the right.
With the game entering the final 10 minutes, there was no sign that a winner was on the cards.
But May, so often Saints' talisman this season, worked a clever one-two with strike partner Steven MacLean on the edge of the box before charging through the Dons defence and into the box before drilling a low shot underneath Langfield.
The 83rd-minute goal sent the Perth support among the 19,057 crowd into raptures but there was soon a loud cry for a penalty from the Aberdeen support as McGinn and Frazer Wright collided at the other end.
Referee Willie Collum waved away the appeal and Saints saw out the remaining minutes to set up what could potentially be the biggest date in the club's history.