The 25-year-old Scot has agreed a two-year contract at Fratton Park, where he will link up with former Robins team-mate Joe Devera.
Ferry told the club website: "Everyone knows how big Pompey are - when you hear some other clubs are interested you don't get as excited.
"I came here with Swindon last year and saw how good the stadium and the fans are. That's a big, big pull.
"I've known Joe for a few years and he came to my wedding. It helps to know someone when you arrive at a new club and he's spoken highly of the place."
Pompey boss Guy Whittingham has overhauled his squad for their tilt at League Two next season.
He said: "I'm amazed and delighted that we've managed to get Simon here.
"He's spent the last four years at Swindon, winning promotion once and getting to the play-offs twice.
"Simon's the kind of player that will work his socks off, so he should prove popular with the fans here.
"We've also got Johnny Ertl and Romain Padovani, so there's some really good competition in the middle of the park."