Richardson guided relegation-threatened Stanley to League Two safety after taking charge at the Store First Stadium in November.
He will now join up with Paul Cook, who he replaced as Accrington manager when he left to join the Spireites.
"Accrington Stanley manager Leam Richardson has left the club to take up a job with Chesterfield," read a statement on Accrington's official website.
"A formal approach has been made for the 33-year-old with a deal agreed between the two clubs after Leam indicated he wished to move on.
"Leam took charge in November and led us to Football League survival last month. All at the club would like to thank him for his service as player, coach and manager over the last eight years.
"Accrington Stanley will now begin the search for a new manager, with current assistant boss Paul Stephenson and first team coach Paul Lodge remaining at the Store First Stadium to handle first team affairs."
Richardson replaces Tommy Wright at the Proact Stadium, after he left the club on Monday as the club made changes to their backroom staff.
"Having worked with him before, I'm delighted to have been able to bring in Leam," Cook told the Chesterfield website.
"He has great knowledge of League Two, as a player, assistant-manager and manager and I'm sure he's going to be a great asset to our club."