Yeoman, recently returned from a loan spell at Dorchester, came on with 15 minutes to go and as the game moved into injury time he pounced to score the winner, driving in a loose ball from eight yards out.
Club captain Lee Mansell returned to the Gulls team in place of Joss Labadie, who started a 10-match ban for biting an opponent, and it he gave the hosts the lead after eight minutes when he headed home a clipped cross from the right by Dale Tonge.
Torquay kept pressing forward, trying to capitalise on a nervous night for Bury's rookie goalkeeper Reice Charles-Cook, but the visitors held firm and could have taken the lead themselves but for a smart save by Martin Rice to deny Danny Mayor.
Bury levelled in the 66th minute when a free-kick broke to Clive Platt inside the Torquay box and he drove the ball past Rice with his left foot.
Charles-Cook saved well from a Mansell volley as Torquay pushed forward in search of a winner, with came courtesy of Yeoman late on.