Steven Schumacher's 10th goal of the season after 21 minutes separated the sides, but Swindon failed to muster a shot on target until the late in the second half when Luke Rooney's free-kick forced Bury goalkeeper Trevor Carson into action.
Schumacher turned in a centre from Joe Skarz for his goal and twice went close to adding more.
Lethargic Swindon missed the influence of former boss Paolo di Canio. The club meet on Wednesday to discuss a replacement.
Miller, joint caretaker with defender Darren Ward, made no attempt to hide his disappointment at Swindon's second poor home result after managing only a draw with Preston.
Miller said: "It was our poorest performance of the season. We did not get going at all. We were simply not at the races.
"Maybe there was some complacency, playing at home to a team in trouble. I don't know the answer to that, but we're very disappointed with the outcome.
"There seemed to be a lack of everything in all departments so the sooner the managerial position is sorted the better for us all.
"You would not think we were second in the league going into the match and with a fantastic opportunity to go top.
"We failed to deliver and there are no excuses for slipping up in the way we did."
