Saturday's eagerly-anticipated fourth-round clash lived up to its billing with Manuel Pascali giving Killie the lead early on, before Dean Keenan equalised.
Craig Bryson then netted what looked like the winner before Allan Russell failed to kill off the tie by missing from the penalty spot, allowing Williams to snatch a dramatic leveller.
Kilmarnock were handed a boost ahead of the tie with the news that goalkeeper Alan Combe had ended speculation about his future by agreeing a new deal to remain at Rugby Park until 2011.
But, despite being available again after suspension, he missed out on the trip to Somerset Park with a neck injury, meaning Damian Rascle deputised between the sticks again.
Killie made two changes from last week's draw against Falkirk, with Danny Invincibile and Bryson drafted into the first team and Jamie Hamill and Gavin Skelton named on the bench.
Ayr made one change from their win over Brechin City in the Second Division, Ryan Borris replaced by Neil McGowan.
Kilmarnock carved out the first real chance of the game when David Fernandez whipped a corner into the packed goalmouth and the ball broke to Bryson who saw his shot deflected over.
The warning was not heeded by Ayr and the visitors took the lead with just nine minutes on the clock.
Fernandez was the provider again with another inviting ball which was knocked across the face of goal by Frazer Wright into the path of Pascali, who deflected home from close range.
Their lead lasted just five minutes before Ayr equalised. Ryan Stevenson's cross was flicked on by David Gormley and Keenan rose above everyone else to nod home from a few yards.
Ayr could have surged into the lead moments later when Chris Aitken's free-kick found Scott Walker at the back post and his header flashed across goal and narrowly wide.
Killie tried to hit back and Fernandez held the ball up well for Grant Murray whose cross struck the woodwork and appeared to catch goalkeeper Stephen Grindlay by surprise before bouncing to safety.
Kilmarnock restored their lead with 53 minutes gone. The impressive Fernandez started the move before Gibson set up Bryson for the shot and the striker produced a lovely finish from just inside the area.
Ayr immediately tried to haul themselves back into the tie and Bryan Prunty was agonisingly close to meeting a Stevenson ball at the near post.
At the other end, Grindlay was called into action to pull off a superb double block, first denying Garry Hay's shot before blocking the rebound from Wright.
Kilmarnock should have killed off the tie as a contest with 20 minutes to go when Hay was fouled in the box by Martyn Campbell and referee Steve Conroy immediately pointed to the spot.
But Grindlay came to the rescue of Ayr again by smothering a poor penalty kick from Russell.
Brian Reid's men refused to give up hope and Stevenson managed to draw a one-handed save from Rascle in the midst of a goalmouth scramble before substitute Williams fired over.
Just when it looked as though Ayr had to admit defeat, Williams delivered a low volley into the back of the net in the final minute to earn a money-spinning replay at Rugby Park.