The 27-year-old, who captained the Millers to automatic promotion back in April, spent a brief loan spell at the Kassam last season where he scored twice in eight appearances.
After passing a medical he has returned to the U's on a three-year deal.
"I had a great time when I was here last year, so I'm looking forward to pulling on the yellow shirt again," he told Oxford's official website.
"Having fought so hard to get Rotherham up last season, I wasn't planning on playing in League Two again. However, I believe that this time next year Oxford will be a League One side anyway and I'm looking forward hugely to helping make that happen and then moving onwards and upwards with the club from there."
Manager Chris Wilder added: "I'm delighted to have been able to bring Johnny back to the club.
"He is a top-quality defender and is the final piece in our jigsaw. He has had a good pre-season with Rotherham so can come in and hit the ground running."