The Robins looked to be heading for an away victory after Shaun Harrad's drilled effort in first-half injury-time.
Jermaine McGlashan almost added a second soon after the restart, but Rovers goalkeeper Steve Mildenhall denied him with a fine point-blank save.
And five minutes into stoppage time Oliver Norburn scored a stunning strike to leave Yates shell-shocked.
"I feel for the boys and I feel for the supporters," he said. "I'm pretty devastated and a bit lost for words.
"Right now I must be the most dejected man in the world. I am probably hurting more than the players at the moment.
"They played really well and made it really tough for us. But it was poor defending from us for the goal. He has hit it from 35 yards and that is not good. We've had 11 men behind the ball in the 95th minute and he's rifled one into the top corner.
"We should have seen the game out but they made it tough for us. But that might be a good point. It would have been a good three points against a team in form, but we haven't done it."
