The Trotters surged into a 2-0 lead thanks to Marvin Sordell and Medo Kamara but Johnnie Jackson pulled one back and, moments after Samuel Ricketts had been dismissed for a second bookable offence, Dorian Dervite levelled.
A Yann Kermorgant penalty completed Charlton's comeback and Bolton were then reduced to nine men late on when Craig Davies received two yellow cards in quick succession.
Defeat left Bolton six points off the play-offs and Freedman chose his words carefully when discussing referee Trevor Kettle's display.
"The sendings off were difficult to take because I don't think Ricketts' first caution was one," said Freedman.
"The referee is only part time and it's difficult for him to keep the intensity, but we put the referee in a position where he had to make the decisions.
"We had a good start and played well, we should have been up by more. But Charlton came at us well and the crowd got behind them.
"I can only control what I can control and pick a team for Huddersfield on Monday. I'm still confident, we've not lost ground.
"I strongly believe that, if we can win a couple of games and put pressure on other teams, we can get back. Today showed it's possible we can be in the play-offs.
"We've now got back-to-back home games and we have been very strong at the Reebok."