In a remarkable 90 minutes that see-sawed one way and then the other, the visitors took a first minute lead through Barry Douglas and continued to dominate the opening exchanges.
But Francisco Sandaza's controversial penalty brought Saints back into the match in the 34th minute and Liam Craig's drive and then a second from former United striker Sandaza gave the home side three goals in 12 minutes and a two-goal advantage at the interval.
The Perth men looked comfortable as time wore on but United then ensured a stirring climax with two goals in just four minutes to rescue a point.
Substitute Gary Mackay-Steven gave the Tangerines hope in the 79th minute and defender Gunning completed the sensational turn around with a header eight minutes from time.
St Johnstone had brought their season alive with a shock victory over Celtic last weekend and their midweek Scottish Communities League Cup win against Livingston but they could not quite make it a hat-trick of victories.
United handed a debut to their new loan signing, 19-year-old Fulham striker Lauri Dalle Valle, and manager Peter Houston would have been thrilled with his side's late recovery.
They could not have dreamed of a better start as they made a stunning breakthrough within the first minute.
John Rankin orchestrated the move with a neat pass slipped in to Johnny Russell on the edge of the area and, when he was allowed to spin, he in turn fed Douglas who drilled in his shot from the left side of the box.
The home side were desperate for an early response but United keeper Dusan Pernis stooped to gather at his near post as Cillian Sheridan tested him with a shot on the turn in the eighth minute.
The goal had clearly buoyed the visitors, however, and they went close to a second two minutes later as Douglas found Dalla Valle 14 yards out only for the Fulham youngster to skew his shot narrowly over the bar.
Douglas was finding himself at the centre of the action and he produced a vital block in his own box before Willo Flood was forced to clear a Jamie Adams shot off the line.
Sandaza was proving a handful for his former United team-mates but he could not direct a curling effort into the top corner after a neat turn on the edge of the box gave him space.
Saints were then handed a route back into the match after 32 minutes when referee Charlie Richmond controversially awarded the home side a penalty.
Russell was adjudged to have fouled former United colleague David Robertson with a high boot and Sandaza, taking over the spot-kick duties from Craig, made no mistake from 12 yards.
Three minutes later Saints were in front. Garry Kenneth was adjudged to have obstructed Jody Morris 25 yards out and, when Morris touched the resulting free-kick to the side, Craig drilled his drive low past the despairing dive of Pernis.
Deep into injury time the Perth men incredibly scored a third. Sandaza hassled Kenneth into losing possession of a bouncing ball inside his own area and the Spaniard swept home past the exposed Pernis from 10 yards out.
Upset United manager Peter Houston left the field at half-time deep in debate with referee Richmond.
Home full-back Dave Mackay saw his shot deflected over for a corner, with Adams then heading Morris's flag-kick just wide.
Cillian Sheridan should have done better than scoop a loose ball well over from 10 yards out.
Houston introduced Danny Swanson and the midfielder at least got on the ball for the visitors and probed for openings.
Russell watched in agony as his long-range header from Paul Dixon's cross from deep slipped narrowly wide.
Richmond waved aside appeals for a penalty when Dalla Valle tumbled easily under a Sheridan challenge.
United then set up a dramatic finale when they pulled a goal back with 12 minutes remaining.
A quickly-taken free-kick saw Russell surge to the byline before pulling the ball back for substitute Mackay-Steven to bundle in from six yards out, despite the best attempts of keeper Peter Enckelman.
United were level four minutes later when Swanson's corner was headed in by Gunning.
Enckelman tipped a Dalla Valle drive over with two minutes remaining.