Andrew headed in a first-half corner and Alan Marriott made a string of saves for Mansfield before Chesterfield midfielder Gary Roberts was shown a straight red card in stoppage time.
Chesterfield had started with the confidence of a team four points clear at the top and Jimmy Ryan forced Marriott to turn behind a 20-yard shot in the fourth minute.
The home side should have turned their dominance into a goal in the 33rd minute but Eoin Doyle shot straight at Marriott and the visitors took advantage when Andrew headed in a corner four minutes later.
Ben Hutchinson almost scored a second when he rattled a post in the 40th minute but the woodwork saved Mansfield when Sam Morsy's shot hit the bar.
Marriott saved twice from Doyle as Mansfield defended bravely in the second half and Chesterfield's frustration boiled over when Roberts was sent off for what looked like a stamp in stoppage time.