The Manchester City midfielder is adamant he was abused by the Russian side's supporters during the Champions League clash on Wednesday at the Arena Khimki.
UEFA has charged CSKA with the 'racist behaviour of their fans' and Romanian referee Ovidiu Hategan could also face sanctions after failing to broadcast an appeal to fans to stop their alleged abuse.
The club have since pleaded their innocence, but Toure has reacted with anger to claims he must have misheard what they claim to have been nothing more than booing.
Their stance has also been backed by CSKA player Seydou Doumbia, Toure's Ivory Coast team-mate, but he feels manipulation is behind his statement.
Speaking to BBC Afrique, part of the World Service, Toure said: "I am not deaf.
"We are all humans. It is not a nice feeling to go and play a football match, to bring joy to the people and to be called a monkey or to hear monkey noises. I don't look like a monkey.
"Other people must have seen it. Doumbia is a young brother. Someone I admire who I have known a long time - we come from the same country.
"I don't want to say things that will put him in trouble but you can see a little bit the manipulation around all this.
"It is so pathetic and so sad to see things (racism) like that. I am ashamed to still have to talk about this subject.
"We know there are very large stakes for the country [Russia] around the World Cup. But let's leave FIFA and UEFA to do their work. UEFA has decided to apply the rules - that's all we want.
"A lot of things have been said about racism. If today an organisation and supporters stray from the straight path or slip up, the goal is to see them straight again."
Doumbia left himself open to question after a confusing account of his post-match quotes.
The 25-year-old was reported as saying his Ivory Coast compatriot "obviously got excited" in relation to him hearing monkey chants.
Doumbia took to his Facebook page to try to clear up the situation, initially writing: "I want to clarify my position after my Ivory Coast team-mate and friend Yaya Toure accused CSKA fans of racism.
"I want to insist that I did not talk to any journalist about these facts so none of the quotes you read in the press came from me."
A few hours later Doumbia added another entry in which he stated: "I want to insist that I did not talk to any journalist about these facts, except just one from Sport-Express, whom I know well, so none of the quotes you read in other publications outside Russia came from me."
The quotes in Sport-Express were the ones reproduced by CSKA's own website and while there may have been slight mistranslations further afield there appears to be no doubt that Doumbia is standing by the words printed by the Russian newspaper.