The Pirates have lost just once in eight games since Ward took charge in December, and this was a fifth away game in a row without defeat.
The first-half was a poor spectacle on the heavy pitch with little goalmouth action.
Rovers goalkeeper Steve Mildenhall saved well from James Constable's 15-yard drive, and a terrific block by captain Tom Parkes diverted Alfie Potter's close-range shot for a corner.
The visitors, backed by nearly 2,000 travelling fans, forged a breakthrough after 55 minutes when Ryan Brunt darted into the area and was brought down by Jake Wright.
Brown had to retake his spot kick because of encroachment, but struck his second attempt in exactly the same place as the first, with Oxford's new goalkeeper Luke McCormick sent the wrong way for a second time.
Substitute Richards sealed victory deep into stoppage time, slotting home under McCormick's body from David Clarkson's quickly-taken free kick.
This defeat was a further blow to Oxford's fading hopes of reaching the play-offs, and Chris Wilder's team have now taken just one point from their last four games.
