The Black Cats were beaten 2-1 at home by Manchester United to remain bottom of the table with just one point from their opening seven games.
Kevin Ball has taken charge for the last three matches following the sacking of Paolo Di Canio and Altidore has been impressed by the interim boss.
But he says the players will not become distracted by talk of who might get the manager's job on a permanent basis and will stay focused on getting Sunderland out of trouble.
"It's been really good to work under Kevin, a really positive experience," Altidore said in The Sun.
"There's been a lot spoken about a new manager but as players we just have to keep going. We can't concern ourselves with decisions being made elsewhere.
"Whatever happens, we're confident we can turn our season around.
"The quality is there - but we have to work for it."