Paul Wotton curled home an exquisite free-kick seven minutes from time after Bees goalkeeper Richard Lee had been sent-off for handling outside the box.
But Brentford had only themselves to blame after squandering a hat full of chances that could have wrapped the game up before the break.
Robbie Simpson saw an effort cleared off the line, Myles Weston saw his shot tipped round the post and Charlie MacDonald fired against the post from close range.
The Glovers took the lead on 22 minutes with a classic counter attack which ended with Oli Johnson bundling home after a mix-up in the Bees defence.
The home side hit back just before the break when MacDonald’s neat turn and volley flew into the top right corner. But another blunder, this time when Karleigh Osborne’s back pass left keeper Lee in no-man’s land, cost them the game.