The Manchester United centre-half was included for the first time since the summer of 2011 for the forthcoming World Cup qualifiers in San Marino and Montenegro.
Ferdinand made no public comment on his recall before telling Roy Hodgson a pre-arranged fitness programme would keep him out this time.
The 34-year-old also made it clear to Hodgson he would like to be considered in future.
Richards, also Premier League chairman, told Sky Sports News: "Rio has been a fantastic servant to England, and to one day say he's going to play and the next day (to say he isn't going to play), I think that does bring the game into a little bit of disrepute.
"Perhaps we need to learn a lesson and do a bit of checking first within the FA.
"It's not good to the Premier League or to Manchester United or Roy Hodgson for one day Rio to say he's going to play and the next day to say he's not. It's something that's not quite right."