One point from a possible six over the Easter weekend means a top-10 finish is now the limit of the side's realistic ambitions with four games remaining.
U's chairman Ian Lenagan may have to concede defeat in his wish for a top-seven berth but the slump has led to increasing pressure on both Wilder and his squad.
"I know there are murmurs and rumblings. I can't do anything about that and I've just got to stick my tin hat on," Wilder told the Oxford Mail.
"I'm not walking away from anything. There's no way I'm going to do that after four-and-a-half years."
He added: "People will talk about where the club is, where it needs to go and what needs to happen. But the club has changed enormously over the last four-and-a-half years and I've been the main part of that.
"I'm not blowing my own trumpet but I, (assistant manager) Mickey Lewis and the staff have done that and are trying to move forward."