Richard Bradley fed Richards inside the box and the striker fired across keeper Luke McCormick and into the bottom corner for his seventh goal of the season.
The result was a blow to Oxford's play-off hopes after Chris Wilder's improving team had registered back-to-back wins last week.
Wilder's decision to recall Deane Smalley backfired with Smalley twice failing to turn the ball in from good positions.
And when he did manage to get a firm shot on target against his former club, after controlling Damian Batt's long ball on his chest, Chesterfield keeper Tommy Lee denied him with a fine smothering save.
In a match littered with free kicks, Oxford got bogged down by the heavy pitch as they tried to mount a response in the second half.
Substitute Tom Craddock went closest to an equaliser with an angled shot that Lee got down to save at his near post.
The U's threw centre half Harry Worley up front late on in a desperate attempt to salvage something, but Chesterfield defended well, with Liam Cooper making one terrific block from Craddock.