The Premier League champions have struggled this campaign under new boss David Moyes and last weekend's 2-0 win over Swansea was their first victory of 2014.
Prior to Saturday's victory, United had lost three successive matches, including an FA Cup third-round home tie to the Swans.
They are currently seventh in the table, 10 points behind league-leaders Manchester City.
Manager David Moyes has been heavily criticised, but full-back Rafael feels the squad need to shoulder the blame.
"We are the ones who have to take the responsibility," said the Brazilian. "I don't know why everyone is looking at the manager. It is the players who have to do the job on the pitch."
In the summer, the club experienced its first major upheaval in 27 years as Moyes succeeded the retiring Sir Alex Ferguson, but Rafael refuses to see that as an excuse for the dip in performances.
"We knew there would be a change," he said. "The manager had just left after 27 years and it was always going to be hard.
"But David Moyes has a strong mentality. He is doing his own job. We have to use that in a positive way.
"We have to show our quality. People are saying United don't have a good squad. We have to prove it is not true."