The U's had failed to win a game in all competitions - their worst-ever start to a season - before Dunne took charge of Saturday's 3-1 success over Hartlepool having replaced John Ward.
And his honeymoon period continued at the County Ground on Tuesday night as former Swindon defender Michael Rose drilled home a 70th-minute free-kick to make it six points from two games in League One.
Dunne said: "To a man we were great but we have to keep building and find ways of combating opponents' tactics.
"We have to learn what we can and cannot do but the players performed to the maximum. My challenge is to make them better players.
"We had an imposing performance from our two centre-backs. Swindon are a magnificent side, which can be driven by their manager into the Championship, but we deserved to win and we edged it."