Kermorgant struck in the 49th minute to give the npower League One leaders their 14th away win of the season.
Both sides had men controversially sent off before half-time, with Charlton captain Danny Hollands shown a straight red card in the 30th minute for his late tackle on Kieran Lee and 11 minutes later, referee Trevor Kettle angered Oldham when he dismissed Jean Yves M'voto for a second booking.
Kermorgant made the breakthrough with a six-yard header at the far post from Rhoys Wiggins' superb left-wing cross.
Oldham's Shefki Kuqi fired off target before Charlton winger Scott Wagstaff became the third sending off of the match, booked for a second time in the 76th minute when he fouled Dean Furman.
Hard-working Oldham pushed hard for an equaliser, but their goalscoring problems continue and they have now lost eight of their last 10 games.