Rovers substitute Williams curled a stunning left-footed strike from the corner of the penalty area five minutes into stoppage time to leave Saints stuck on a negative points total.
Rickie Lambert predictably had a hand in both Saints' goals as he returned to haunt his former club by laying on the first for Adam Lallana with a fine inside pass and the midfielder applied a sublime finish.
However, Chris Dickson's pace caught the home side napping before the break when a long free-kick forward had goalkeeper Kelvin Davis in no-mans land and the on-loan Charlton striker stroked the ball into the net.
With just 40 seconds on the clock in the second half, Lambert headed against the underside of the crossbar and over the line to put Southampton back in front.
Jo Kuffour equalised less than 10 minutes later when he converted from Carl Regan's purposeful run but with the game heading for a draw, Williams cut in from the right-wing and unleashed a thunderbolt.