Lee Brown's penalty 10 minutes into the second half, and a second goal deep into stoppage time by substitute Eliot Richards did the trick for the visitors.
Oxford have taken one point from four games over the past fortnight, dealing a severe blow to their play-off aspirations.
"We had the best chances of the game but haven't taken them," Wilder said. "It's just as disappointing as last week when we lost at Southend.
"In a tight first half we had the best chances, and I thought we started the second half really well.
"It's just a bit of quality in and around the final third that is the difference.
"And that's the difference between players staying here and players moving on.
"We dominated that game but I can't keep saying that. It's the big moments we're not getting right."
"My skipper (Jake Wright) has held his hand up for the penalty. He's made a bad decision.
"The second goal at the end was because we had six up front and were trying to get an equaliser."