Rickie Lambert's spot-kick earned Southampton a 1-0 win at Villa Park, condemning Villa to a third straight home defeat in the league and sending them into the bottom three.
Referee Mark Halsey awarded the decisive penalty after Jay Rodriguez tumbled under an Enda Stevens challenge, and replays vindicated the Villa players who argued there was no contact.
Lambert told Sky Sports: "It's nowhere near it. You've got to be 100% certain to give a penalty, and when Mark sees that I think he'll be embarrassed by it.
"He can't guess - you certainly can't do that in a game of that magnitude. You have to be 100% right, and to me that was a guess."
Another home defeat was hardly what Lambert wanted on the back of Tuesday's Capital One Cup humiliation at Bradford.
He said: "It's disappointing, but that's the way things seem to be going for us at the moment - we don't seem to be getting the little breaks.
"We had loads of chances, but if we keep that effort up and that commitment - that's the way to play the game of football.
"We're getting kicked all over the place at the minute, but you keep battling through and meeting the challenge head on. The crowd again were excellent, right behind us."
On the prospect of recruiting this month, he said: "It's tough - we know the situation at the football club. The chairman's been very good with me, we'll see what happens."