Gabriel Agbonlahor seized upon a Lee Cattermole error after 15 minutes to score the only goal of the game and despite plenty of endeavour there was no way back for the Premier League bottom club.
"We started well," Poyet told Sky Sports. "We were controlling possession and creating chances. But we made a mistake that, unfortunately, changed completely the game.
"They shouldn't happen because they are so simple. The pitch is in good condition and the ball rolls.
"But everything happens for a reason and I don't know if the reason is that it is going to be more difficult than I thought to get out of this situation.
"Always something happens and I'm getting a little fed up."
The home supporters at the Stadium of Light roared their team on but the tension was evident and the Sunderland boss believes that could also be a problem for his team.
"There was plenty of time but they (the fans) are not used to watching a team passing the ball and controlling the game," added Poyet.
"If you make a pass it is going to be a little bit easier, but they want to do everything in a hurry.
"The relationship after the mistake between the players and the fans wasn't great. There were nerves in the stands and that transmitted to the players, who started to play too quick.
"They wanted to do everything in one minute. Everyone wanted to score the equaliser and be the hero. This is a team game. Team games mean we have to all play the same game.
"Until we understand that, it's going to be more and more difficult."
Emanuele Giaccherini did have a goal disallowed in the second half and while Poyet was not happy with that decision and others, he refused to allow it to deflect from his team's performance.
"That's a good excuse," said Poyet. "The problem today was that we were not good enough."