The in-form Baggies head to the Stadium of Light at the weekend looking to record a fourth straight Premier League win as they look to maintain their place in the top four.
Swedish international Olsson insists Steve Clarke's side deserve to be in such a lofty position, though they understand they face a tough encounter against the Black Cats.
"I think going to the Stadium of Light is always a difficult task, and I think, as long as we know that as a group of players, I'm not bothered about the sorts of pressure that people outside the club are putting on us," he told the club's official website.
"We need to realise that we are a good team and we're not going to apologise for where we are in the league.
"We deserve every point that we have earned so far and though we're going to Sunderland and we want to win, we also know how hard it is.
"We know what we need to do and hopefully if we put in a performance like we have been doing then we have a chance of getting a result.
"We need to keep hold of the things we do well, like our shape, and then build from there.
"We need to be humble about the task and we're a team that needs everyone to be performing well in order to get a win."