Having come through the ranks, Caulker was tipped to become Tottenham's long-term replacement for Ledley King.
The 21-year-old made 28 appearances for Villas-Boas last season and even earned an England cap, but the Tottenham manager surprisingly sold the centre-back to Cardiff for £8.5million this summer.
Villas-Boas stands by his decision, but he admits it was not one that he took lightly.
"Yes, it was one of the tougher decisions I have made here," the Tottenham manager said.
"At that time we were short at the back. We had just lost Jan (Vertongen) during the Hong Kong trip but we felt the price was right and it was an opportunity that we had to take."
It was not just the big money that was on the table from Cardiff that persuaded Villas-Boas to sell Caulker.
After winning his first cap against Sweden, the defender was keen to make sure that he was in Roy Hodgson's thoughts for next year's World Cup.
To do that, he would have to be playing on a regular basis, and despite the departure of William Gallas this summer, that was not something that Villas-Boas could offer.
"I can understand why people were surprised (that we sold him) because he was home-grown and he was a player who did well for us, but in the end we have great competition for places and we decided for Steven to move to another club," the Spurs manager said.
"He has high expectations towards his career, towards a first-team place and we decided to allow that transfer to happen."
Villas-Boas will see Caulker for the first time since he sold him on Sunday when Tottenham travel to Cardiff for a potentially tricky away game.
Tottenham scraped two 1-0 wins and lost to Arsenal in their opening three games, but they have started to show clear signs of improvement against Norwich, thanks largely to the introduction of Christian Eriksen.
Eriksen, who moved from Ajax to Tottenham last month for £11.5million, set up two goals on debut against the Canaries and he came off the bench to score a stunning goal against Tromso in midweek.
The Dane has provided Spurs with the effective link between midfield and attack that has been missing since Rafael van der Vaart returned to Hamburg.
Villas-Boas clearly thinks he has got one of the buys of the season.
"For someone so young it's extraordinary that he has such tactical awareness," Villas-Boas added.
"We all know that adapting to the Premier League is difficult and his game against Norwich shows we have a mature player who can adapt quickly. He gives us that dangerous final pass, which he has shown with his assists against Norwich."
Villas-Boas will be without Aaron Lennon because of a foot problem while there are doubts over the fitness of Danny Rose, Younes Kaboul and Mousa Dembele after they all suffered injuries in the 3-0 win over Tromso.
Still, the influx of summer signings means Villas-Boas is far from short of options for the trip to south Wales.
With the international break coming up, the Portuguese has set his side the target of chalking up two league wins in six days - against Cardiff and Chelsea next Saturday.
"We want to win both of them, that would put us in a very good position," the 35-year-old added.
"We want to keep continuing this run of results we have been getting. That would be great motivation for the team."