Baines has played left-back in the successive 2-1 defeats to Italy and Uruguay and his performances, particularly in Thursday night’s loss, have been criticised.
But the Everton defender does not feel the back four should take the bulk of the blame for a start to the tournament that has left the team on the brink of an elimination at the group stage.
When asked if he thought the defence was the weaker part of the unit after the Uruguay defeat, Baines said: "Not particularly.
"I don't think we have given up too many chances in the two games, but when we have, we have been punished.
"The tournament has been a bit like that. There have been a lot of goals so far and we'll have to look at it in the next couple of days."
Baines felt England were the better side against Uruguay but that they were punished by a player capable of capitalising in the smallest of margins.
"We got ourselves back in the game and got a bit of momentum going," Baines said.
"With that momentum it felt like we may have been able to go and win it ourselves but we were punished in the end. It is just disappointing.
"It was difficult. I thought we were probably the better team over the course of the whole game but we have been punished by good goals.
"Suarez is a top player. He didn't put himself about as much as he normally would but looked like he was always waiting for that moment, that opportunity to pounce, and he did.
"You are at the top level of the game and it is small margins - we haven't been outplayed in any of the games, we have just come up short maybe."