The Black Cats were under the cosh from the off, as the Canaries enjoyed 80 per cent of possession in the opening 10 minutes.
Chris Hughton's side continued to press forward and created several openings before Robert Snodgrass fired the hosts into a 20th-minute lead before Alex Tettey's powerful volley doubled Norwich's advantage soon afterwards.
Defeat keeps Sunderland 18th and three points adrift of safety, but Poyet has no fears about his or the Black Cats' future in the top flight.
"I try to do my best but I'm not worried," he said.
"If we keep looking at the rest and that results went good for us today, we forget about ourselves.
"You need to make sure you do your bit because that is what you are getting paid for."
An impressive opening spell from Norwich left Sunderland on the back foot and Poyet believes the home side were deserving of the lead.
"I think the first half-an-hour especially, we were second best in every act," he claimed.
"That's how football goes against us right now. Tettey, he doesn't score too many, but he hits one from 45 yards [sic] and gets it in the top corner.
"I don't believe too much in luck, I think to have the luck you have to be the better team and sometimes you get what we did today."
Poyet made two tactical substitutions five minutes before half-time but refused to single out Ki Sung-Yueng and Jack Colback, the players hauled off, and was left upset by a slow start.
"They made me make decisions I don't like - I don't like changing players in the first half. It wasn't a case of one or two players, It was seven or eight but you can't change seven or eight," he added.
"I think it is very disappointing when you start a game with the importance of today the way we did.
"We started badly and we weren't getting close to any of the action. I think it was a mix of being bad on the day and Norwich being very brave."
Despite defeat leaving the Black Cats in the dreaded drop zone, they do have games in hand and Poyet understands the importance of the Premier League run-in.
"We have to make sure we put points on the table now," he claimed.
"We have 10 games - I think we are the only team to have this quantity of games - and we know we need to get over 36 or 37 points.
"We need to win four from 10 or win three and get some draws. We have to play better than today, that is for sure."