The result seriously dented the U's play-off hopes after they had chalked up wins over Torquay and Barnet in the previous two games.
And Wilder was left cursing their failure to convert the first-half chances.
"We were so wasteful in the final third in the first half," he said. "That first 45 minutes was possibly the most we've been in control here all season.
"I couldn't believe it at half-time that we weren't out of sight. We had 20 crosses, which is a ridiculous number, but couldn't get on the end of one.
"The game was lost for us in the first half. We needed to get something out of that and we didn't. It drives me absolutely crackers because coming off at half-time, that should have been done and dusted.
"Richards has just one chance, and takes it. That's why he's so sought-after by clubs.
"I'm not critical of the enthusiasm or the competitiveness of my players, but they were just not clinical when they needed to be.
"It was the same with the goal we conceded. If you don't defend properly and switch off, and allow an opponent to run off the back of one of our defenders, you are going to get punished.
"Then in the second half Chesterfield grew into it, and we faded."
