The Brazil boss, who spent seven months at the Blues before being sacked in February 2009, returns to West London on Monday when his side face Russia in a friendly.
During his time at the Bridge, Chelsea were fourth in the Premier League and still in both the FA Cup and Champions League, and Scolari feels he should have been given more time to make an impact.
"I believe my time at Chelsea was not as bad as people have been talking about," he said.
"The numbers show that, the games we won. I don't think it was that bad."
Asked whether he was given enough time, Scolari added: "I don't think so but it was a decision from the management.
"Everywhere I work I am just a member of staff and if somebody thinks I am not doing a good job, they can sack me.
"That happens everywhere. I accept that. I did enjoy my time here at Chelsea and I was happy while I was here. It was one of the great experiences I have had in my life."
Chelsea defender David Luiz said Scolari is still held in high regard at the club, saying: "Mr Scolari is an excellent professional. He has been a coach in many clubs all over the world and also national teams.
"I'm sure he is happy to come back here because regardless of what happened here he has very good memories.
"He has got history here as well so I believe what he wants here is to put the cherry on the top of the cake by winning the game at the stadium where he was manager.
"When somebody doesn't do well everybody outside always talks about the negatives not positives.
"What I have heard, having asked around, is Scolari had a great time, regardless of what happened."