Michael Laudrup's side have now gone eight Premier League games without a win and were beaten 3-1 at home by Tottenham on Sunday to leave them three points above the relegation zone.
Michu, Michel Vorm, Jonathan de Guzman, Jose Canas, Pablo Hernandez, Nathan Dyer, Garry Monk and Jonjo Shelvey are all on the sidelines but Williams is convinced they should still have improved on their meagre tally of just three points from a possible 24.
"We're working hard, we're talking a lot about it," he said.
"We have been playing against some top teams, but still teams we expect to get something out of. That's not an excuse.
"We also have injuries to key players maybe but again that's not an excuse.
"You can see in the way we are playing that confidence isn't a problem, it doesn't feel that way anyway, we are still getting on the ball a lot, and when teams are lacking confidence they tend to shy away from it.
"We're trying to give it a go but I can't work out what's going wrong.
"I've never conceded so many goals in my Swansea career but we're not giving ourselves a chance.
"Maybe if Wilfried (Bony) scores and we go one up it could have been different. I take full responsibility."
Swansea's next four games are against Fulham, West Ham, Cardiff and Stoke, and Williams admits that run of fixtures could make or break their season.
"It is crunch time, we said how important the Christmas period was for us and we didn't really perform and now it's another crucial period," he added.
"If we have a few good results in the next four then we'll be okay but if we don't then the pressure is on.
"We have played some tough teams that are scrapping for their lives as well."