Altidore's disallowed goal was the main talking point to emerge from their 3-1 loss to Arsenal as referee Martin Atkinson blew for a foul on Sagna against the American, who did not hear the whistle and played on - finding the back of the net.
Altidore admits that he was hugely disappointed not to have opened his account for the season.
"Of course I thought that was my first goal in the Premier League. You don't get chances like that in this league very often and I thought I had my goal," he told the Sunderland Echo.
"It was very disappointing. I didn't hear the whistle or anything. I don't think Sagna heard anything because he kept going and I think it was (Kieran) Gibbs who tried to get it out of the goal so he obviously didn't hear anything either.
"The keeper came out and played the ball like it was live.
"It crossed the line okay as well, so it's disappointing."
Sunderland find themselves bottom with just one point from their opening four games, but Altidore insists that nobody is panicking about their situation within the club.
"You don't get time in the Premier League before people are talking about relegation and stuff," said Altidore.
"But everybody is calm still.
"We still have plenty of time, but we understand that we have to start getting points.
"I thought the second half was good, but we can't wait for teams.
"You can't wait in this league, there's too much quality.
"You have to start the game like we did in the second half.
"Arsenal had a lot of chances in the first half, but we put them on edge in the second half. If we had started the game like that, I think we would have got something.
"We need that intensity because that's how we're going to get more chances.
"We're not going to tap the ball around teams like Barcelona.
"We're going to get chances by being aggressive, by being tenacious and we have to keep doing that."