Poyet, who replaced Paolo Di Canio as manager at the Stadium of Light, took charge of his first Premier League game but watched on as four second-half goals consigned his side to defeat.
Having only collected one point from their opening eight games, Sunderland remain rooted to the foot of the table, with Poyet admitting that the poor results have affected his team mentally.
"We analyse all of the problems, technical, physical and tactical, but of course there is a problem we need to address and the most important is the mental side," he told Sky Sports.
"Nowadays, in football or in any sport, you need to be strong mentally, this is not about one person, this is about everybody at the club, especially the 11 out there on the pitch. So maybe there is more work to do mentally than the rest."
Poyet also bemoaned the second-half performance of his team, having conceded three goals in the space of seven minutes as Swansea eased to victory.
"It was a completely different team in the second half. I think it's been happening for a while, it's a team that when it's in the game and fit enough, it does things naturally," he said.
"But when something happens that goes against the team, it changes completely.
"Like I said, I learned a lot today. I learned a lot from individual players, from situations, from reactions and I'm sure I can put that right. In certain aspects, we need to improve.
"Normally, you don't get any luck. We had a couple of half-chances in the first half, one from a corner and a couple of breaks.
"We didn't get enough of the ball in the box and when something happened in the other half, there would be a deflection or something and it's going in.
"That's how it is, we're not going to change it apart from working to make sure we do the same things for the 90 minutes and win a football game."