Right-back Matt Fish netted the only goal of the game for the hosts in the 24th minute with a fierce left-footed strike from the edge of the box that cannoned in off the underside of the bar.
Accrington could only muster half-chances from Peter Murphy and Francis Jeffers in response and still sit just one place and two points off the bottom two following their defeat.
"We feel we should be coming home with at least a point. It's frustrating because I want to praise the lads, especially in the second half I thought we were outstanding," said Richardson.
"The goal has killed us and it's a great goal to be fair but the onus is on us, we've got six games to go that are must win.
"It's about results and I'm disappointed but the lads put a massive shift in. You could see the belief because we are a good team and we've given a good account of ourselves.
"But compliments aren't going to keep us out of the relegation places, we need to win games. Coming off the back of a couple of good results and performances we really fancied our chances.
"We were full of confidence playing against the top of the league and at times we played them off the park."