Sherwood publicly criticised his players following Spurs' 4-0 defeat at Chelsea and that was followed by a 3-1 home loss to Benfica in the Europa League.
England striker Defoe joined Toronto from the North London club earlier in the month and has urged Tottenham to stick with Sherwood.
He said: "Tim has made improvements since he came in. The training has gone up a notch and is sharper.
"Andre Villas-Boas was good but we were not scoring many goals. Tim came in, and in the first few games, you could see the difference in how we were playing.
"We were creating a lot more chances and were very direct. We were playing the Tottenham way. Tim is a nice man and he really wants to do well but it's going to take time."
Sherwood has been tipped by many to lose his job at the end of the season, following his criticism of the side last week. Defoe though, thinks that would be a step backwards.
He added: "You could see in his eyes in his TV interview after the Chelsea game how much it means to him.
"Tim's been there and done it and he knows his football. It's important to have someone like that at your club.
"I think he is the right man for the job."
Tottenham entertain local rivals Arsenal at White Hart Lane on Sunday.