The League Two side took the lead just after the half-hour mark when Tom Pett took advantage of a mix-up in the Hammers' defence.
Keeper Adrian came sliding out of goal but failed to gather a header back from Aaron Cresswell, leaving Pett to slot home.
West Ham boss Sam Allardyce made 11 changes at the break but the hosts went two up when a trialist turned neatly before shooting past Jussi Jaaskelainen into the bottom corner.
However, experienced winger Matt Jarvis pulled a goal back with six minutes remaining when heading home unmarked from close range following a Guy Demel cross.
And Jarvis turned provider in injury time, crossing from the left byline for Downing to nod in at the far post.