In an incident-packed match, Ramires and Christian Benteke saw red as Chelsea came from behind to win 2-1 at Villa Park.
"That was a hard one to take, the way that game was going - it was there for everyone to see why we lost that game," Lambert told Sky Sports.
When asked to clarify, Lambert added: "The one man in the middle, the decisions, his decisions were really strange."
John Terry was booked early in the second half for bringing down Yacouba Sylla when he was about to run clear and Lambert was insistent that should have been a red card.
"John Terry, a wonderful player, but nobody would have caught Sylla, nobody," he said.
"My team were fantastic, we gave everything we had and we did not deserve that.
"They were fabulous, they really were and you can see by the crowd - the way they played, they got better and better and did not deserve to lose."
There was also an incident when Chelsea claimed a goal as they thought Demba Ba had scored, only for Ashley Westwood to clear off the line - but Lambert says that was the right decision.
"The whole ball is obviously not over the line, he got that right but he has 90 minutes to get a few right but I thought that some of his decisions were bizarre," he said.
Villa remain five points clear of trouble, knowing they could still be relegated.
"Everybody has a lot of hard games, we have the 40-point mark and some teams are playing against each other and have hard fixtures so let's see what happens," he said.