Two goals from Carlton Cole were not enough to overhaul Southampton, whose 4-0 win against Coventry ensured they held onto second spot in the Championship.
Instead, the play-offs beckon for Sam Allardyce and his side, who will face Cardiff in the first leg on Thursday.
Allardyce had planned to be in Las Vegas on May 19 watching Amir Khan's next fight, but now he hopes to be at Wembley instead as the Hammers attempt to secure promotion the hard way.
It is their home form which has proved costly - West Ham have won a club record 13 games on the road but dropped 28 points on their own turf to slip out of the top two.
Unfortunately for them, a third-placed finish means they will have the dubious 'advantage' of being back at home for the second leg a week on Monday.
The Hammers were desperate for an early goal to set the nerves jangling down at St Mary's, and they should have grabbed one in the fourth minute only for Ricardo Vaz Te to somehow head wide from all of three yards out.
And any scrap of pre-match optimism around Upton Park quickly evaporated as the news Saints were making short work of the Sky Blues filtered through.
The atmosphere suddenly went flat but the hosts continued to attack and Henri Lansbury blew another chance to put them in front when he blazed a volley over from 12 yards.
Matt Taylor then clipped the crossbar with a curling cross which left Hull goalkeeper Vito Mannone bewildered.
And in the 35th minute West Ham did get their noses in front when Cole met Mark Noble's corner with a downward header which Paul McKenna could not keep out on the goal-line.
They should have gone further ahead before half-time but Vaz Te fired across goal, Kevin Nolan made a mess of a header and Noble's fierce shot was blocked.
But three minutes after the break the lead was doubled when Cole controlled Lansbury's cross on his chest and expertly dinked the ball over advancing Mannone.
Hull had barely threatened Rob Green's goal, but they were gifted a consolation with 10 minutes left when Guy Demel's ludicrous header let in Corry Evans to nod the ball home.
For West Ham, their consolation is the play-offs as they attempt to bounce straight back from last season's relegation.