Luke Howell snatched the Dons' match-winning goal on 85 minutes when he volleyed past goalkeeper Peter Gulacsi.
Peter Levin kept the pressure on the home defence with a low drive past the far post but just before the break Tore Andre Flo was upended by Josh Gowling, who was yellow-carded for the offence.
But Hereford almost took the lead 10 minutes into the half when Jennison Myrie-Williams flicked the ball past Willy Gueret who had raced off his line but the winger's shot was cleared away by Shaun Cummings at the foot of the post.
Striker Steve Guinan however had a gilt-edged chance to break the deadlock but he lifted Marc Pugh's cross over the bar.
But Howell made no mistake when presented with the late chance.
League One play-off semi-final fixtures: First legs: Scunthorpe v MK Dons (Friday 8 May), Millwall v Leeds (Saturday 9 May, 12.45pm).
Second legs: Leeds v Millwall (Thursday 14 May), MK Dons v Scunthorpe (Friday 15 May).