Scolari, who became the first Brazilian coach in the Premier League, lasted just seven months in charge at Stamford Bridge after arriving in West London in 2008.
The 65-year-old admits he was upset at how things worked out for him at Chelsea, but conceded he would consider returning to manage in England.
"I was upset by the way my departure from Chelsea happened," Scolari told the Guardian.
"I wish my work at Chelsea could have lasted longer, especially because I was prepared to spend at least two or three seasons in London and experience the Premiership.
"I was being honest with the club in everything I did. We were doing reasonably well on the pitch.
"A few players did not agree with the decisions I had to take for the good of the team. I heard them complaining and saying: 'I do not play in this or that position.' Never mind that one of the players I had put into the first team was actually one of the league's top-scorers at that point."
"If I had to return [to England], I would do it. Chelsea are a spectacular club and I still want them to do well.
"The supporters are amazing and even though things did not work out I still remember how nice the fans were to me."
Meanwhile, Scolari has tipped England to do well at next summer's World Cup finals.
"England are one of the teams who could shine at the World Cup," added Scolari.
"Roy [Hodgson] is doing a great job in bringing new players to the team and is overseeing a change in generations.
"England have a good team and I think they are much better technically these days.
"People say I'm a specialist in defeating England but the victories with Portugal against them in Euro 2004 and in Germany in 2006 were on penalties, really tight games. You know how tense shoot-outs are.
"I think England are missing the confidence to take that step that finally puts them among the top four."