The Terriers were rescued by Adam Clayton's late penalty to cancel out Rob Hulse's opening goal for the home side.
But Southern's controversial red card, following a challenge with Michael Morrison, was the main talking point.
"I've seen it on the DVD and it's two committed players going in 50-50 on a wet pitch," he said. "He clearly got the ball first.
"But the referee made a quick decision. It's changed the outcome of the game.
"It was a shame as it would have been a great game 11-11. The conditions were there to pass the ball around and it had the makings of a great game but then it was spoilt.
"Even Morrison said to Keith that you got the ball first. He has not jumped off the ground. We will be appealing. It does feel like a win for us though scoring so late."
Charlton manager Chris Powell added: "It's not a sending off. It's a honest challenge and both didn't pull out."