Substitute David Syers headed home in injury time as Rovers stepped up their bid for promotion from League One.
The visitors took a third-minute lead when Billy Paynter's effort came back off the post and defender Rob Jones headed in the rebound from eight yards.
Oldham then took control and made several chances before levelling after 27 minutes, Youssouff M'Changama's 25-yard drive flying through the grasp of Rovers goalkeeper Gary Woods.
The busy Woods made a fine save to deny Jean Yves Mvoto and, in a close-fought second half, both sides were on course for a point, but Rovers left-back Tommy Spurr swung in a deep 91st-minute cross and Syers headed past Oldham goalkeeper Dean Bouzanis from close range.
Saunders said: "We aren't stringing passes together as much as I'd like, but we're getting results and that's all that matters. I'm learning about the players all the time and we will carry on building and getting better over the next few months.
"I've given the players a short-term target and a target for how many points we'll need by the end of the season.
"I won't go into detail, but the first priority is to get through the Christmas period in at least the same league position as we started.
"You could say scoring in the last minute is a bit of luck, but our full-back was still getting forward and we showed the desire to take all three points.
"Oldham are a good side on their day and I expected chances at both ends, which is what happened, so I'm delighted to win."
