The Dutch winger wrongfully picked up centre-back Jack Stephens from a 31st-minute corner, only for the Saints man to head home after being given the freedom of Villa Park.
“We’ve got El Ghazi marking their centre-half when he shouldn’t be,” Smith said.
“John McGinn (should have been) but, after he went off, I believe it should have been Douglas. The message never got on. El Ghazi ends up marking him and the lad scores a free header.”
Danny Ings scored a double to take his Premier League tally to eleven, with Jack Grealish grabbing a fine consolation for his fifth of the season.
Smith added: “Players have to take responsibility at times, and I have to take responsibility as well. We’ve conceded seven from the last two home games.
“We were very poor defensively against Leicester and, today, they were ore individual mistakes that’s for sure.
“First one, Bjorn gets done over the top. The second one, it’s a corner and a free header and the third one, Fred must get closer to the lad and it ricocheted off Marvelous. There’s certainly elements we need to get better at defensively.”