Adam Clayton and Sean Scannell were on target for Huddersfield, who will be at home to Premier League Wigan in the fifth round.
However, Lillis may not be in charge for that match because Huddersfield have been given permission to speak to Coventry boss Mark Robins about their managerial vacancy.
Having taken an early lead at Leicester, Huddersfield were quickly pegged back, but Scannell got the winner in the second half after a fine counter-attack.
Lillis said: "This was a fantastic evening for the club.
"The big thing was to get the lads into the dressing room at half-time and make sure they continued to believe in themselves. That was the big word - belief.
"We are very happy, of course we are. It is a great feeling.
"I don't really know anything about the managerial situation, although I'm told the club have made an approach for someone.
"But we've got a job to do and it was a case of making sure the players focused on what they had to do.
"It is great to be in the fifth round of the FA Cup, and at home, and it will be great to pass it on to the new man if he is here.
"We have had a lot of adversity since I came into the job three weeks ago and we had a young side out there tonight, but they did us proud."