Adams took charge of the Canaries last month after his success within the club's youth set-up, which included a 3-2 win over the young Blues at Stamford Bridge to win the FA Youth Cup.
But things have not gone well since he took over from Chris Hughton, and the Canaries head to west London on a run of five straight defeats that has dropped them into the bottom three.
"I have very good memories (of Stamford Bridge)," Adams said.
"The last time I was there as a coach was the Youth Cup final and we were successful. If there are any coincidences or traces of luck then hopefully that will happen.
"It is exactly the same, everyone outside of Norfolk doesn't give us a chance on Sunday. Chelsea are going for the title and we are in the bottom three and it doesn't take a genius to work out who the neutral will be fancying to win the game.
"It was exactly the same with the Youth Cup final, nobody really gave us a chance against a big and expensively-assembled Chelsea side but it is very similar to now, if you get the players right and get them believing they can go there and cause an upset they can."
Adams accepts his players are at a low ebb but he is expecting a reaction from their 4-0 defeat by Manchester United last weekend.
"It is very easy to get into the mentality when you are on a losing run. They say winning can become a habit and we have to make sure losing doesn't become a habit," he said.
"We have to prepare the players for if we go a goal down, does everything go out of the window or do we stick at it and make sure we have got an alternative option or an alternative plan. When you have lost games and you are at the bottom of the table it is tough."
Adams' opposite number Jose Mourinho has come under fire for his tactics in recent weeks, leaving him to wonder if any manager will ever be free from criticism.
He added: "Jose is one of the best there is, one of the best there has ever been.
"It will be a huge privilege to stand in the dugout next to someone of that calibre but for 90 minutes I'm not concerned about what is happening in the technical area, I'm interested in what is going to be on the pitch.
"He has gone from being a tactical genius to being questioned. I don't know how you can question Jose Mourinho for what he has won and achieved in the game but that is the nature of it.
"If you get a negative result and the finger will be pointed and you will have to answer questions, that applies to everybody."