Forest took the lead courtesy of a Matt Smith own goal - though the Oldham striker was a thorn in Forest's side for the rest of the afternoon.
Robbie Simpson struck twice within four second-half minutes to turn the game in the visitors' favour and after Forest skipper Danny Collins saw red for a foul on Smith, Jose Baxter made it 3-1, and Billy Sharp's goal was no more than a consolation.
McLeish said: "We were woeful in the second half. We were sloppy in everything we did and paid the penalty for it.
"We warned the players at half-time that they had to be professional in the second half but we had to be a lot better than that.
"We gave the ball away cheaply, didn't close them down quickly enough and it was all very half-hearted.
"Their big lad up front (Smith) caused mayhem in the second half and we didn't deal with it. Our centre-halves had to be ugly and rugged to deal with the threat.
"I was pretty pleased with the first-half performance but we looked a little bit frightened after the break. We allowed them to press us and it unnerved us. Why the players allowed that to happen is a mystery to me."
Oldham manager Paul Dickov is hoping for a big pay day and a fourth-round draw with his former club Manchester City.
"With my City connections it would be nice to meet them in the next round. Financially it would mean a lot to the club and it would be great for me on a personal level.
"If we can make some money in the next round, you never know - the chairman might let me bring one or two players in as well.
"But for the time being we just want to enjoy today's game because I am very proud of my players for the way they come back into this game after losing an early goal."