Until the on-loan Tottenham defender connected with a David Cotterill free-kick it had looked as though Paul Dickov's men were going to be taken to extra-time by Keith Hill's gallant League Two side.
There was little doubting Rovers deserved a first win under Dickov, though, for it was they who had done the majority of the running in the largely lifeless 88 and a half minutes that preceded the goal.
Although often missing an extra touch of class in the final third, they were twice denied by the woodwork and also ran into an in-form goalkeeper in the shape of Josh Lillis.
Towering striker Chris Brown saw a first-half header cannon onto the bar, while after the break he was denied by the former Scunthorpe keeper.
Lillis also impressed as he tipped over a James Coppinger piledriver and, even when a Coppinger shot was too hot for him to handle in the 57th minute, he still somehow managed to tip it up onto the bar.
Rochdale's best work came from Matty Lund who twice went close with swerving efforts, but Dale were undone when Khumalo went up for a free-kick alongside centre-half partner Rob Jones and got the final touch as the pair jumped together.