The 2004 win over Millwall has been followed by disappointing final defeats to Arsenal and Chelsea, along with two semi-final losses, most recently to Manchester City in 2011.
Of the current squad, only Ryan Giggs, Paul Scholes and Darren Fletcher have winners' medals, and Smalling says the Red Devils boss gave his side a firm talking to ahead of Saturday's tie at West Ham.
"The manager has made the point to us that we have not done too well in the last few years," Smalling said.
"We need to make sure we get back on track in the competition.
"The league is going well and we're into the Champions League knockout stages, so we want to do well in this as well."
While the continued success of both the Premier League and Champions League have downgraded the FA Cup's status within the English game, Smalling still views it as a special tournament.
He said: "The FA Cup has always been a massive competition.
"You've got the situation where lower league teams have the chance to play some of the top sides. It can be a real adventure."
United top the league with a seven-point advantage over rivals City, but the 23-year-old admits the game at West Ham will not be easy.
"It wasn't the easiest draw we could have chosen. Sam Allardyce has got a good team together and done very well," Smalling said.
"The fans really get behind them, especially at Upton Park, which will make it more difficult for us. But it is always somewhere I have enjoyed playing and it will be a good atmosphere."