The Hammers had travelled to Germany on the back of four successive friendly victories but they were no match for the hosts in the first half.
Dani Schahin gave Mainz a ninth-minute lead when he collected Nicolai Muller's precise pass and fired past Jussi Jaaskelainen.
The home side doubled their advantage in the 21st minute through Muller when he raced clear down the middle after combining well with Benedikt Saller.
Mainz, who are further ahead in their pre-season preparations, continued to dominate before the interval and Muller almost made it 3-0 on the stroke of half-time but Jaaskelainen diverted his shot behind.
West Ham improved after the interval and they were back in the game in the 47th minute when Joe Cole volleyed home after Ricardo Vaz Te had nodded back Mark Noble's free-kick.
Modibo Maiga then went close to equalising for the visitors before Mainz restored their two-goal cushion in the 74th minute with Shawn Parker netting from the spot after James Tomkins was penalised for a foul on Yunus Malli.
Sebastian Polter then volleyed home a fourth goal for the Bundesliga side from Parker's centre four minutes later.