The Chelsea defender is well aware of the quality Bale possesses, but is not losing any sleep over whether the former Tottenham man features or not.
Bale played a full part in Wales' final training session on Monday, but manager Chris Coleman has said he will not start in Cardiff.
Ivanovic said: "Bale is one of the best players in the world, everyone knows that, but it is not just him. There are other great players in the Wales squad and they have played really well in some games.
"Bale is one of their main players but we do not know whether he will play or how long he will play, but we have to be ready to face a side with him or without him, and we are."
"It's difficult to stop him, he is a player who is always hard to mark. It is a pleasure to face the best players, but for us it is important to play as a team as one individual cannot make that much of a difference over a game."
Ivanovic also insisted the 6-1 drubbing which Serbia dished out to their Group A rivals in Novi Sad last year will have little bearing on Tuesday's clash.
He said: "In football it is always difficult and they will be motivated looking back at the 6-1 defeat the last time we played. We know that.
"They are at home, it has been a long time since the last game and it will be tough, but we are well-prepared."