The Frenchman produced a number of impressive saves in the Capital One Cup clash at Carrow Road but failed to deal with Grant Holt's header near the end of the match, allowing Simoen Jackson to score the winner.
Norwich had drawn level just moments earlier through Jan Vertonghen's own goal, with Spurs earlier taking the lead thanks to Gareth Bale.
Villas-Boas admitted it was a concern to see his defence again crumble in the dying minutes under pressure, saying: "It is obviously something that is in our mind at the moment.
"It is obviously difficult but the further you train these situations sometimes the worse it becomes.
"You cannot match the emotion and concentration players face during the game. We obviously have to do better and try to improve things for the next game.
"It's a pity to see the game slip from our hands. We had a good performance."
Asked about Lloris, Villas-Boas said: "He did extremely well in the game. He had a good solid performance. It is difficult for goalkeepers in situations like these."