The 23-year-old joined the Millers on loan until the end of the season last month, making eight appearances as Steve Evans' side secured promotion to League One.
Full-back Skarz , who turned out 145 times for the Shakers, will pen a two-year deal at the New York Stadium.
A statement on Bury's official website read: "Our thanks and best wishes are extended to Joe as he prepares to make the move to the New York Stadium."
Evans told Rotherham's official website: "I think he's been absolutely sensational since he's joined us.
"He's salt of the earth, and he wants to come and work. He wants to improve, and for me he will play in the Championship one day, and he's only 23 now, so we hope that it will be with Rotherham United."