Scolari, whose side take on England in a Wembley friendly on Wednesday, was approached by the Football Association in 2006 with the opportunity to take over from Sven Goran-Eriksson after the Germany World Cup.
But the Brazilian, then contracted to Portugal, pulled out of the running - at the time blaming media intrusion - and has admitted the decision still weighs heavily on his conscience.
"Of course it hurts," said Scolari, who went on to manage Chelsea two years later.
"It hurts a lot because I would have loved to have been the manager of the England team. Who wouldn't? It's a wonderful national team.
"When I was invited to be national coach of England, when we sat down I still had a contract with the Portuguese national team and I was not willing to break that contract.
"It was my duty to fulfil it. Imagine what it would have been like when we played each other in 2006 and I had signed that contract with England."
When asked about any future plans with England, Scolari added: "Who knows what will happen. One day, maybe.
"I wish all the best for the English players and their manager on their path to innovation."
Roman Abramovich appointed Scolari as Chelsea manager in 2008, but after the Blues' mid-season collapse he was fired after just six months in charge.
Refusing to criticise the Russian, Scolari said: "I had a marvellous time in London. I loved it.
"I hope I will have the opportunity tomorrow to talk to Frank Lampard and Ashley Cole.
"What is happening right now (at Chelsea), I don't have any opinion on."