Holt headed home the Canaries' third just before the break as his side beat Michael Laudrup's Swans in a thrilling encounter in South Wales.
The Norwich captain scored only his fourth goal of the season but he says that he is happy as long as the team is performing.
"People go on about a goal drought but it wasn't as if the chances have been coming," said Holt. "I hope I've been doing a good job for the team and that people around me can see that.
"If I was missing chances week in, week out, then I'd take any criticism coming my way. But they've been falling to other people and they've been putting them away, so it doesn't bother me."
Norwich play host to Aston Villa in Tuesday night's Capital One Cup tie and it will be the first time that Paul Lambert has returned to Carrow Road since quitting at the end of last season.
Holt believes that his side should be confident of continuing their impressive form, saying: "It's been a long time since Norwich had a really good cup run. But we are at home and will be expected to go out and give a performance.
"We expect to win the game. It's a fantastic opportunity to get into the semi-finals. We're in great form and we'll take the same mentality into that game.
"Nobody can criticise what Paul did for the club and I'm sure it will be a bit strange for him coming back and being on the other side of the dug-out.
"He'll be listening to the fans that have sung his name and hailed him. It will be weird, but we'll get on with it and try to beat him like any other manager."
Don't miss any live and exclusive football coverage this festive season. Click here to visit Sky Sports' Festive Calendar