Johnson, who was re-appointed as boss on Monday, watched on as Williams got on the end of Dean Parrett's left-wing cross to nod home from close range in the fifth minute of added time.
Adam McGurk opened the scoring for Rovers with a superb free-kick, curling the ball past goalkeeper Rene Gilmartin. in the ninth minute.
Yeovil, with three players making their home debuts, equalised after 19 minutes through one of the newcomers. Joe Edwards picked up the ball 22 yards out when a corner was only partially cleared and drove the ball past the left-hand of keeper Paul Rachubka.
Johnson's side continued to have the better of the first half but came under increasing pressure in the second. They almost conceded in the 57th minute when Mustafa Tiryaki shot wide and there was another let off in the 84th minute when Robbie Weir's effort struck a post.
Referee Simon Hooper pulled a muscle in the second half and had to be replaced. That meant seven minutes of stoppage time and Williams pounced to seal the victory.