Mark Halsey awarded a spot-kick for an Enda Stevens challenge on Jay Rodriguez, but replays showed the striker was quick to go to ground under little or no contact from the Villa defender.
Rickie Lambert ignored the home protests to bury what turned out to be the winner, and Adkins was in no mood to apologise afterwards.
He told Sky Sports: "It's a big talking point. For me the big decision is the one on Jay Rodriguez, I think that's what he's given it for.
"There's no contact, for me, Jay has moved his leg out of the way - and it's just as well as there would have been contact.
"We don't advocate diving whatsoever - we've got to make sure everyone is well aware of that - but they say it evens itself out and we could have had seven or eight penalties this season. We've got one here and Rickie Lambert's put it in the back of the net."
Victory took Saints above Villa in the table on the back of a four-match unbeaten run in the league.
Adkins said: "It's a good three points away from home against a team close to us in the division. It's a clean sheet, too, which is important.
"We could have had the game out of sight in the first half. In the second half - credit to Villa - they came at us and we've had to show a lot of character to weather the storm."