The Uruguayan inherited a squad in some disarray when he replaced Paolo Di Canio at the Stadium of Light helm and will head into his first game at Swansea on Saturday with the Black Cats already six points adrift of safety.
However, American striker Altidore, while knowing how important the club's short-term goals are, insists there is a bigger picture.
He told the Journal: "I think it kind of stunts when you are trying to build something because of the pressure of staying in the league.
"Every year, there are different types of teams in the Premier League just trying to stay up.
"If we can just focus on building a good team and try to leave the rest of the stuff out, it will be good for us. Hopefully the new manager can help us do that."