Latching onto a clearance down the left wing, Potter raced past Leon Legge to slot the ball calmly into the far corner in the 84th minute, the first goal the U's have scored against Gillingham in seven games.
In the first half United keeper Luke McCormick excellently denied first-half substitute Bradley Dack a goal with his first touch after the midfielder had been played in by Deon Burton.
Full-back Liam Davis responded two minutes later, travelling with the ball from inside his own half down the left channel before firing in a shot which Nelson saved well with his legs.
The Gills brought on Danny Kedwell and new signing Antonio German but both failed to make an impact and the hosts have now won just two of their last nine games on home soil whilst Oxford earned just their second win in eight games.
