The 25-year-old will put pen to paper on a two-year deal when his contract with the Addicks expires on July 1.
Sullivan spent the last couple of months of the season on loan with AFC Wimbledon and was offered a permanent deal, but he has opted for a move to Fratton Park.
He told Portsmouth's website: "I'm very excited to be joining Pompey.
"I was down at Wimbledon for the final part of last season. It's a great club and I owe them so much.
"But this is a great opportunity to come to a big club at the start of a new project. As soon as I heard of Pompey's interest, it was an easy decision. This is where I wanted to come.
"I have been at big clubs like Brighton, Millwall and Charlton, but never cemented myself as number one.
"Pompey is definitely the biggest club of my career and I am coming here with one aim - to play and help the club get promoted to where they need to be.
"I'm coming with the attitude that, at the age of 25, I want to play, although I know it's not going to be as easy as that.
"I have competition at the club and so it's very much up to me to do the business and prove myself."