Forster-Caskey fired the visitors in front with a brilliant curling shot from the corner of the box after 21 minutes and Potter doubled the lead with a low finish after cutting in from the left 11 minutes later.
Rovers improved marginally after the break but lacked the penetration to break down a well-organised visiting defence and there were few anxious moments for Oxford goalkeeper Ryan Clarke.
Home boss Mark McGhee threw on an extra forward in Eliot Richards five minutes after the break, but his best efforts came from distance and never troubled Clarke.
Oxford passed the ball more sharply without creating many clear chances themselves and looked an efficient unit.
Rovers barely managed a worthwhile first-half effort and when Joe Anyinsah did head home after 16 minutes the flag was already up for offside.
