Djamel Abdoun put the Championship side ahead via the penalty spot in the 12th minute after George Moncur had fouled Paterson in the box.
Paterson slotted home on 65, 71 and 79 minutes and Andy Reid scored in injury-time to see their side easily progress to the fourth round.
Sam Allardyce faces fresh questions over his future as West Ham manager as his unrecognisable side were routed.
While Forest's win might go down in some quarters as a cup upset, once the team-sheets were announced the bookmakers had already installed the hosts as odds-on favourites.
Allardyce, with injury and suspension problems and a Capital One Cup semi-final in midweek, handed out a total of five debuts, three from the start and two more off the bench, among nine changes to his team.
It was a gamble that backfired horribly as the relegation-battling Hammers simply had their confidence further assaulted by lower-league opposition.
The young rookies - Seb Lletget, Danny Whitehead, Callum Driver, Reece Burke and Blair Turgott - can hardly be blamed for the humbling result, but Allardyce may not be as lucky.
His future may detract slightly from Paterson's efforts, but it should not - with the winger winning his side's spot-kick and then converting three fine goals of his own.
The opener came after just 12 minutes as Paterson beat Moncur with some neat footwork before being tripped by the midfielder. Referee Martin Atkinson pointed to the spot and Abdoun dinked the ball down the middle with a cheeky finish.
West Ham's most experienced players were stationed on either wing, with captain-for-the-day Stewart Downing left and Matt Jarvis right. The Hammers looked for the pair with every attack and in the 23rd minute, the visitors went close to levelling.
Reid's foul gave Ravel Morrison a chance to test his free-kick skills from 20 yards out and his bending shot would have crept under the crossbar had Karl Darlow not responded with a fine save.
At the other end, there was a big penalty appeal soon after when Driver went in hard on Greg Halford as he looked to convert from close range, but the defender got a piece of the ball.
Regardless, West Ham were clinging on - goalkeeper Adrian tested first by Jamaal Lascelles' long-ranger and then by Halford's downward header.
Three minutes before the break, Forest missed two more chances to double the lead. Reid's clever through-ball deserved a better finish than Eric Lichaj could provide and, when Adrian's save came back out, Henri Lansbury shot over from an inviting position.
After the break, Downing's shot deflected over and Whitehead followed up with a wild attempt of his own.
Allardyce threw on more youth, handing debuts to Burke and Turgott and also sending on Mathias Fanimo.
The visitors might have equalised before the hour mark had it not been for Darlow, who saved low from Lletget while unsighted and stood firm to Morrison's near-post drive.
That was as good as it got, though, with the game ended as a contest on 65 minutes when Halford held the ball up well before teeing up Paterson to find the bottom corner.
Paterson's second came just six minutes later as West Ham's frailties continued to be exposed. Lansbury played creator this time, squaring the ball for Paterson, who stroked home confidently with a first-time effort.
By now, Forest were all over their opponents - Lansbury, Paterson and Abdoun taking turns to torture their young markers.
Paterson's hat-trick duly arrived and there was one final insult when Reid smashed home Abdoun's lay-off in injury time, leaving West Ham thoroughly humbled and with plenty to ponder.
|Nottingham Forest||Team Statistics||West Ham United|
|11||Shots on Target||5|
|9||Shots off Target||0|