The decision by referee Steven McLean to award a penalty after Marc McAusland was adjudged to have bundled over Dale Carrick just after half-time, which saw the defender sent off and Jamie Hamill score from the spot proved the decisive moment in the game.
Earlier, Steven Thompson had given Saints a first-minute lead before Callum Paterson levelled for Hearts four minutes later.
After reviewing the penalty incident, Lennon felt that the between McAusland and Carrick occurred outside the box and was not enough to warrant a foul.
"I'm angry and I'm angry at the fact it was a game that was there for the taking, certainly in the second half," he said.
"A big, big decision has gone against us within the first five minutes and it has cost us dearly.
"I'm angry with the decision, I'm not angry with the performance because I thought even with 10 men, the energy they brought and the opportunities we created, we had Hearts on the back foot.
"You can certainly see this young Hearts side are maturing, they brought a lot to the game but it was a big, big decision that has cost us.
"I've watched it four or five times and I'm still inconclusive whether there is any contact at all.
"If you're asking me 'is it inside the box?' No, it's not.
"Sometimes they go for you and sometimes they don't. I certainly thought it was a game-changer in terms of trying to control the game."
Hearts manager Gary Locke was pleased with the character his side showed to come back after falling behind so early in the game.
"I thought the character was there in abundance again, we've shown it all season," said Locke, whose side also won 2-1 at Ross County on Saturday. "We've had a lot of lows but this week has been great for us.
"I think you can see that it gives everyone a lift, not just myself and the backroom team but the players and the supporters.
"It has been a very difficult season for us but I have been over the moon with the way the players have performed."