All four goals came in the first half with Luke Freeman cancelling out Bakayogo's opener before Ben Gibson put Rovers back in front and Bakayogo extended their lead with his second just before half-time.
Tranmere went ahead in the 10th minute as Bakayogo forced the ball home after Jean-Louis Akpa Akpro's initial effort was blocked.
Stevenage equalised two minutes later as Freeman curled a left-footed free-kick from the right corner of the penalty area past Owain Fon Williams, but Gibson replied four minutes later after another Akpa Akpro shot was blocked.
Akpa Akpro should have made it 3-1 himself just after yet he missed the target, however, seven minutes from half-time Bakayogo scored his second after a horrendous mix-up between David Gray and goalkeeper Steve Arnold.
After the break Robin Shroot wasted Stevenage's best chance of getting back in it as he fired wide after receiving the ball in a glorious position in the heart of the Tranmere box.