Riether only recently agreed a permanent deal at Craven Cottage having spent last season on loan at the club, meaning Schalke's chances of bringing in the former Germany international were always slim.
However, the form displayed by the former Freiburg player persuaded Schalke to chance their arm but a difference in valuation of the player is reported to have prevented a deal.
General manager Horst Heldt spoke to Ruhr Nachrichten on Tuesday and appeared to admit defeat in their attempts to sign Riether.
"We are starting the campaign with the squad we currently have," he said.
"We will only look at incoming transfers again if our first-team squad is hit by injuries. If not, we will not spend further (on new players)."
Riether will now remain at Fulham, having signed a contract keeping him at the club until 2015.