The American striker endured a forgettable debut campaign at the Stadium of Light, netting just twice in all competitions.
He and Sunderland had been hoping for much more, with a £6million transfer pushed through on the back of an impressive strike rate in the Eredivisie with AZ Alkmaar.
It has been suggested that his inability to provide value for money could lead to his time on Wearside being cut short after just 12 months.
Altidore is, however, convinced that he will come good, as he prepares for World Cup duty with the USA, and says his boss feels the same.
He said in the Sunderland Echo: "You are going to go through tough times and in a career in soccer, a lot of it is mental.
"You have to stay strong at the tough moments because there are going to be a lot of them.
"Poyet said, 'I know it didn't go well for you, but I believe in you and I want to see you back here'.
"So it was positive."
Altidore added on the need to reflect on the past but drive on into the future: "It was tricky but it was also good to sit back and see what I could do better.
"I feel like I improved, even though I wasn't in the team.
"The bigger picture is we stayed up in the Premier League.
"When it comes to the World Cup, the season is behind you and all that matters is what you do going forward."