Two days after Kiwomya was sacked the Magpies secured just their third league win of the season to move off the foot of Sky Bet League One.
Headers from Danny Haynes and Yoann Arquin, either side of Adam Rooney's leveller for Oldham, gave County a 2-1 lead before McGregor struck and while Jonson Clarke-Harris pulled one back, the hosts held on for the win.
County took the lead after 19 minutes when Haynes powerfully headed a Greg Tempest cross past goalkeeper Mark Oxley at the near post.
Jack Grealish came close to increasing the Magpies' lead in the 23rd minute, his header from an Alan Sheehan free-kick going inches wide of the target.
But on the stroke of half-time Rooney nodded home James Dayton's fine cross.
Arquin's header restored the hosts' advantage before McGregor's powerful finish after 68 minutes gave the Magpies some breathing space after Haynes had missed an open goal.
Two minutes later, Latics forward Clarke-Harris rifled in a 25-yard free-kick, but he missed a good headed chance late on as County clung on.