The shoot-out went to sudden death and Drew Talbot converted the crucial spot-kick after visiting keeper Tommy Lee denied Oldham's Anton Rodgers.
Oldham went ahead after 21 minutes when David Mellor's left-wing cross found Danny Philliskirk, who turned sharply and drilled inside the far post from 10 yards.
Chesterfield were level within seven minutes as Tendayi Darikwa planted in a close-range header from Talbot's excellent first-time cross.
The Spireites almost snatched the lead, with the dangerous Darikwa firing wide from a good position
But Oldham dominated the second half and Lee rescued his side three times, saving well from Philliskirk, Jonson Clarke-Harris and Jonathan Grounds.
Lee also denied Adam Rooney at the death and it was Chesterfield who held their nerve from the spot to move within one round of Wembley.