After 23 minutes midfielder Oliver Norburn hit a low shot straight at Krysiak from 25 yards, but the keeper somehow allowed the ball to pass between his legs and trickle over the line.
Krysiak was then left clutching at thin air by Brown's stunning left-footed drive from the corner of the box eight minutes from time, which saw the ball rocket into the roof of the net.
Promotion-chasing Exeter had plenty of possession, but were unable to create any clear-cut chances.
Krysiak did well to tip a first-half header from Tom Parkes around a post and Rovers emerged worthy winners.
The result took Rovers nine points clear of the drop zone, having been bottom of the table when John Ward was appointed manager in December.
For Exeter it was a rare blemish to their excellent away record.
