Benedikt Howedes headed the German side in front four minutes before half-time at the Karaiskakis Stadium and, after Djamel Abdoun equalised, Klaas Jan Huntelaar grabbed what proved to be the winner.
The Holland striker then missed a penalty but his side nevertheless made a winning start.
In a quiet first 15 minutes, Paulo Machado had the only effort of note but Kyriakos Papadopoulos blocked.
Huntelaar threatened for Schalke before the two teams traded free-kicks, Jose Holebas shooting wide for the hosts before Schalke's Tranquillo Barnetta struck one into the wall.
Rafik Djebbour headed Machado's corner wide before Pablo Contreras blocked Christian Fuchs' shot at the other end.
Captain Howedes was booked for a foul on Francois Modesto but Schalke pushed forward once more, Jermaine Jones shooting over.
Papadopoulos saw a headed goal disallowed but defensive colleague Howedes broke the deadlock three minutes later.
Jefferson Farfan's shot was deflected wide and the Peru forward took the corner himself, picking out Germany international Howedes to head home from close range.
Fuchs drew a sharp save from Balazs Megyeri early in the second half, Barnetta shooting over on the follow-up, but the home side then seized the momentum.
Abdoun was denied by keeper Lars Unnerstall and Joel Matip cleared Djebbour's follow-up, but the equaliser was not long in coming.
Djebbour laid the ball square to Abdoun, who slotted home in the 58th minute.
But Schalke immediately retook the lead, Huntelaar latching on to Lewis Holtby's pass and firing low past Megyeri.
He should have made it 3-1 on the hour after Kostas Manolas brought down Barnetta in the box, but smashed his spot-kick against the left-hand upright.
Ariel Ibagaza curled a shot over for Olympiacos and they went close twice in the 81st minute, Machado's header blocked by Fuchs before substitute Leandro Greco headed Machado's corner wide.
Jones was forced off by injury late on, replaced by Marco Hoger, but his side held on as Djebbour shot over from six yards deep in injury time.