The Gunners had been looking for a response to Saturday's lacklustre 1-0 loss at Norwich, but failed to produce one as Arsene Wenger watched on from the stands, serving out his European touchline ban.
Schalke now overtake Arsenal at the top of Group B, and were good value for their victory, secured by second-half goals from Klaas-Jan Huntelaar and Ibrahim Afellay.
The Gunners must now regroup ahead of Saturday's visit of Premier League strugglers QPR and next week's Capital One Cup tie at Reading, but this poor showing will have done little to lift the mood ahead of Thursday's Annual General Meeting.
Schalke beat Bundesliga rivals Borussia Dortmund away last time out, and were looking to recover from the disappointment of seeing 10-man Montpellier snatch a late 2-2 draw from their previous European tie.
The visitors started brightly, and had a penalty shout on 13 minutes following a quick break.
Jefferson Farfan played in Dutch winger Afellay, who fell over the outstretched arm of Vito Mannone.
There looked contact from the Italian keeper, whose handling error gifted Norwich their winning goal on Saturday. However, Swedish referee Jonas Eriksson waved away the appeals and instead brandished a yellow card to the Schalke player.
German Lukas Podolski was Arsenal's main outlet down the left and his low centre flew back across the six-yard box, just behind Aaron Ramsey.
Gervinho had been preferred in the central striker's role to Olivier Giroud, and was sent clear - but lacked support and rather than trying to go it alone for goal, was eventually hounded off possession as he tried to hold the ball up.
On 32 minutes, German centre-half Per Mertesacker made an important block to charge down Farfan's fierce drive.
There was another scare for the hosts just before the break when Japan full-back Atsuto Uchida got clear down the right and crossed for Huntelaar at the near post, but the usually reliable Dutchman swept a poor effort wide.
Schalke made a change for the second half as American midfielder Jermaine Jones, who had a loan spell with Blackburn in 2011, replaced Marco Hoger.
The Germans should have taken the lead when Farfan charged past Andre Santo down the right and crossed back to the penalty spot, but captain Benedikt Howedes blasted over.
Slowly Arsenal started to find their passing game, and Gervinho was unfortunate to lack support in the middle as he pushed on down the left to cut the ball across an empty six-yard box.
The hosts, however, still had plenty of defending to do as Schalke remained dangerous down both flanks.
On 70 minutes, Gervinho went into the referee's book for going down under a challenge from Jones in the box.
The Ivory Coast forward was soon replaced by Giroud.
It was Schalke, though, who eventually got the breakthrough on 76 minutes.
Arsenal failed to clear their lines, with Afellay heading the ball back through the centre of the defence and into the path of Huntelaar, who made no mistake from 10 yards.
The Gunners made a double change when Podolski was replaced by Andrey Arshavin and full-back Carl Jenkinson came off for 17-year-old Serge Gnabry.
It was, however, game over with four minutes to go when Schalke broke quickly down the right through Jones, and his low centre was swept home by Afellay.
The inquest started among the disappointed Arsenal players - a debate which no doubt will be continued by disgruntled shareholders on Thursday.