Jussi Jaaskelainen was the match-winner for West Ham, saving penalties from Felipe Santana and Axel Borgmann in the shoot-out after earlier producing a fine stop to deny Eric Maxim Choupo-Moting with his legs early on in normal time.
Choupo-Moting also fired against the base of the post in the first half before Marvin Friedrich hit the crossbar in the second period as the Premier League side rode their luck.
Carlton Cole had their best chance to win it but instead it was left to Kevin Nolan, James Collins, Elliot Lee, Diego Poyet, Andy O'Brien, Dan Potts and Sebastian Lletget to do the business from 12 yards.
The Hammers face Malaga on Sunday, while Schalke take on Newcastle.
Summer signings Carl Jenkinson and Aaron Cresswell will play against the Spanish side having been rested for this game, along with Adrian, Winston Reid, Mark Noble and Matt Jarvis.