The 23-year-old midfielder gave Spurs a huge boost on Wednesday night by coming off the bench to score the winner in a 2-1 victory at Fulham.
This brought an end to a run of four games without a win, which included a 6-0 defeat at Manchester City that resulted in some heavy criticism of boss Villas-Boas.
However, Holtby claims that Villas-Boas is "very close to the team" and that the club's players are unified in their determination to succeed.
"We are sticking together," said Holtby. "We are not discussing things that are happening outside - that is not our job.
"Our job is to get points to win games and make the fans happy. That is our main objective.
"We have to do our talking on the pitch. We were responsible for that disaster at Manchester City but the way we responded after that has been good. We have to carry on doing that."
Tottenham invested heavily in the summer following the sale of Gareth Bale to Real Madrid, with critics suggesting the new players have under-performed.
In addition, Villas-Boas had a public spat with a journalist in the press conference after the 2-2 draw with Manchester United after the latter had written a damning article about the former Porto boss.
Speaking about Villas-Boas' influence specifically, Holtby said: "He bonds us together and is very close to the team. Everyone accepts him and his way of playing football.
"There is a lot of negativity coming from outside but we are all sticking together. Bringing this bad mood into the team will not help. We are positive and we know that the season is long.
"We all stick behind the gaffer. We give our best.
"Of course we had a tough time in November but last night we got a good result at a difficult place."
Tottenham travel to Wearside to face Sunderland on Saturday and Holtby is eager for his side to make it two wins in a week.
"Now we need another three points from the Sunderland game," he added. "There are games every three days in December so we have to be up for it every time.
"Last season, we did really well during this time and got ourselves to the top of the table.
"The fixtures we have are very important now and there is a possibility of getting a lot of points."