The former Spurs midfielder's position is coming under increasing scrutiny after a run of one win in six games but he was in typically abrasive mood ahead of the Monday Night Football clash against Sunderland, live on Sky Sports 1HD.
A number of supporters expressed their irritation at Sherwood watching from the stands in last weekend's defeat at Liverpool.
"You are damned if you do and damned if you don't," he said. "If Jose Mourinho sits up there (in the stands), he's fine.
"Nigel Pearson at Leicester has sat up there all season. But I am realistic to know it's Tottenham, you are under the spotlight no matter what you do.
"You've got a rookie manager coming in here, won the first eight games or whatever it was and everyone thinks it's easy. I didn't, I was realistic, I knew there were going to be some downs.
"It was a case of just cracking on. But whatever decisions I make are my decisions. If I'm going to go down, I'll go down my way.
"It's just one stage at a time and I've got six cup finals between now and the end of the season, and it's really important to me that we show the fans something to make them optimistic about next season."
Sherwood wants to see some fight from his players against Sunderland, but was quick to deny talk of fisticuffs in the away dressing room at Anfield last week.
"It was incredible the way that it gathered so much pace," he added. "I was getting family and friends texting me saying, 'Why did you do it?'. They didn't say, 'Did you?', it was all 'Who did you hit?'."
Some players took to social media to defend their manager, who added: "I don't know because I'm not on Twitter - that's why I don't speak with Gary Lineker.
"I didn't know if the players had come out or not, I hadn't heard that but I am not surprised. They know I want the best for the club.
"And that helps them out because it means I want the best for them, but I am not going to suffer fools. That is the way I manage."
Watch Tottenham take on Sunderland at White Hart Lane live on Sky Sports 1HD from 7pm on Monday.