The 25-year-old midfielder, who was named in the PFA League Two team of the season, has signed a three-year deal.
Manager Chris Wilder said: "I am really pleased to welcome John-Joe to the club.
"We have had to work really hard to get this deal done so a lot of credit must go to the Chairman on this. We know he turned down three clubs in League 1 and others in League 2 to come here and it is great that he wants to be part of what we are building.
"He can see the ambition of the club with all of the changes being made to the stadium and the infrastructure, the way we finished last season and the signings we have made this summer and that is a big attraction to players who want to be successful.
"He is a quality player, he offers a big goal threat as his total last season proves and we are really looking forward to working with him."
O'Toole spent just one season at Rovers after a loan spell in 2013. He has also had spells at Colchester, Sheffield United and Wealdstone after starting his career at Watford.