Ian Black gave the home side the lead with a deflected drive in the 28th minute and in the 56th minute he was crunched in a tackle by Joe Ledley to earn the Hoops midfielder a red card.
Two minutes later the Gorgie men scored a breakaway second through David Templeton, but there was much more drama to come.
After a penalty claim was ignored by referee Craig Thomson, when Jambos substitute Ryan Stevenson, on for Black, looked like he might have used an arm in the box, the raging Lennon was dismissed from the dugout after aggressively complaining to fourth official Steven McLean and linesman Graham Chambers.
The incident is sure to spark more controversy and debate in light of the Irishman's recent spat with the Scottish Football Association and referee Dougie McDonald after the official rescinded a penalty award at Dundee United and then lied to Lennon about how he had come to the decision.
Effectively playing with four strikers, Celtic were intent on taking all three points back to Glasgow and they started well. In the seventh minute Hoops skipper Shaun Maloney picked up a Georgios Samaras pass and moved to the edge of the Hearts box before firing straight at Jambos goalkeeper Marian Kello.
Moments later Celtic survived a goalmouth melee following a Rudi Skacel corner and then, in the 13th minute, Stephen Elliott crashed a header off the crossbar from 10 yards after Templeton volleyed towards goal from the right-hand side.
The challenges and tackles continued to fly in as both sides battled for dominance but in the 28th minute the home side enjoyed a slice of fortune when they took the lead through Black.
The former Inverness midfielder took a short free-kick from Skacel and let fly from 30 yards, watching as the ball took a deflection off Daniel Majstorovic before looping over Fraser Forster in goal.
The Hoops fought back and there was controversy in the 34th minute when striker Gary Hooper was booked for simulation by referee Thomson after going to ground when rounding Kello in the box.
There was barely time to take a breath and Skacel tested Forster with a free-kick from distance, the Hoops keeper grabbing at the second attempt.
Four minutes from the break Maloney dispossessed Hearts skipper Marius Zaliukas on the byline, allowing Anthony Stokes to set up the unmarked Hooper but his low drive from 16 yards was blocked by Kello's legs.
It was a let-off for the Edinburgh side men and later, at the other end, Zaliukas tried to make amends when he climbed to meet a Skacel corner but headed over the crossbar.
Celtic went hunting for the leveller with some enthusiasm at the start of the second half and the Tynecastle men found themselves with their backs to the wall.
Hearts took some time to emerge from their own half but when they did, the looked threatening.
The game boiled over again in the 56th minute when Ledley was handed a straight red card by Thomson after a tackle on Black which left the Hearts player requiring treatment.
The Celtic bench was enraged but two minutes later there was more trouble for Celtic when Skacel raced down the left and crossed for the unmarked Templeton to slide past Forster from 12 yards.
Moments later, Kello made a point blank save from Hooper before Lennon introduced Cha Du-ri and Niall McGinn for Stokes and Mark Wilson.
In the 70th minute there were loud claims for a Celtic penalty when Ryan Stevenson made an intervention in the box with what might have been an arm but referee Thomson played on.
Lennon complained long and loud to the fourth official McLean and linesman Chambers, and was ordered by Thomson to leave the technical area.
Six minutes from time McGinn hit the outside of the post from the byline and while the visitors continued to fight, there was no way back.