The Scot has been in charge at Villa Park since the summer of 2012 and spent his first season at the club waging a battle against relegation from the Premier League.
A run of five wins in the final 10 games was enough to keep Villa in the top flight, but another fight to avoid the drop appears in store this season.
Villa's four-game losing streak was halted by a 1-1 draw with Swansea City on Saturday, but Lambert's side are only four points outside the relegation zone.
Just two victories from 10 home games have also led to murmurs of discontent and Lambert is aware that he cannot take his job for granted.
"I've got a good relationship with Randy, but that doesn't mean I'm immune to it," he said.
"I always have that fear. I always had that fear as a footballer and I always do when I'm the manager. I think you've got to have that.
"It's been the same at every club I've managed.
"I don't think it will get any tougher than what happened last year and he was great with me then. I spoke to him last night and he's been good now as well.
"We have a plan. Randy has been great on that front.
"But I feel we're a million miles from where I want to go. I also know it is a long road but it is a challenge I relish."
Lambert acknowledged the importance of picking up a point against Swansea, particularly in the absence of influential trio Christian Benteke, Ron Vlaar and Jores Okore.
He added: "It's a game that you'd love to win, but a point stops that sequence of losing, which was vital for us.
"We're without big players, and it's a lot for us to carry. We're a young squad and when you lose people like Benteke, Vlaar and Okore, it's a big chunk of our spine.
"We (his management team) have only been here 17 months and I knew the first few years was going to be really tough, but we're halfway there when you look at the table. It's a big point."