Norwich's victory at the Liberty Stadium moved the Canaries up to eighth in the Premier League table.
The Canaries had enjoyed the better of the opening half but trailed at the interval after Danny Graham had coolly slotted the opener from Nathan Dyer's pass in the 23rd minute.
But Norwich were ahead within six minutes of the restart as Holt's header deflected in off Ashley Williams and Anthony Pilkington's strike ricocheted in off Neil Taylor, with the striker slotting the third from a tight angle just after the hour mark.
Holt blotted his copybook by pulling back Williams to give away a late penalty, which Graham converted, but Norwich held on to become just the second side, after Manchester United, to beat Swansea on their own turf this season.
Gylfi Sigurdsson had the chance to give Swansea an early lead after confusion between Kyle Naughton and Canaries keeper John Ruddy, but Russell Martin got there in the nick of time to block the shot.
Norwich settled the better of the two sides, pressing high up the pitch and enjoying some joy with Simeon Jackson running at Angel Rangel down the Swansea right.
But it was Swansea who took the lead against the run of play in the 23rd minute.
Sigurdsson linked well with Dyer and the winger played in Graham for the striker to coolly net his 11th goal of the season.
Holt was heavily involved as Norwich sought a rapid response. The striker, having headed over the bar moments earlier, accused Williams of attempting to elbow him in an aerial challenge before Holt found himself facing the same accusation from Rangel as tempers flared.
Norwich's best chance of the half fell to Andrew Surman, but the midfielder's diving header from Pilkington's brilliant cross was straight at Michel Vorm.
Swansea should have doubled their lead with half-time looming after Dyer and Scott Sinclair had teed Graham up for a tap-in, only for Elliott Ward to brilliantly deny the striker.
Having been unfortunate to trail at the interval Norwich turned the game on its head as they grabbed two goals within six minutes of the restart.
Firstly, Holt headed home via a deflection off Williams after Surman had done well to keep Martin's skied attempt at a volley in play on the byline.
And another deflection helped the visitors claim their second as the excellent Elliott Bennett teed up Pilkington and his shot struck Neil Taylor, who had put through his own net in Swansea's last home game against Chelsea, and deceived Vorm to put the Canaries in front.
At the other end Steven Caulker struck the post from a Sigurdsson corner in between the two goals, while Sinclair was guilty of a poor headed miss from a Dyer centre.
But Swansea were being overrun and when they ceded possession in midfield, Bennett made the most of a three-on-two situation and fed Holt for the striker to beat Vorm from a tight angle.
Naughton survived penalty appeals for handball after blocking a Dyer shot and a cross from substitute Wayne Routledge and then came with inches of making it 4-1 as his rising angled drive was tipped onto the bar by a full-stretch Vorm.
Swansea's desperation to find a route back into the game was leaving them short at the back and they were almost punished when Jackson dispossessed Leon Britton to play Holt in. The striker unselfishly set up substitute Wes Hoolahan, but Vorm made a spectacular save to tip his effort over the bar.
Having done so much to put his side in control Holt then needlessly gave away a penalty for pulling back Williams, with Graham scoring from the spot to set up a tense final few minutes.
Swansea laid siege to the Canaries goal as the game entered five minutes of stoppage time but Graham could not complete his hat-trick as he skewed wide before Ruddy made a superb save to keep out Caulker's point-blank header and secure the three points.