Serbia lost to their arch-rivals on Friday, and Caldwell admits that will have hit them hard ahead of Tuesday's World Cup qualifier.
Caldwell wants Scotland to capitalise on that defeat and bounce back from their own loss to Wales last week.
"It is probably the equivalent of Scotland v England and if we had lost that it would have been a big blow," Caldwell told Sky Sports News.
"We have to try and use that to our advantage and start the game better than we did on Friday, and try and get the fans on their backs and grow into the game from there."
Caldwell insists that, despite their struggles in recent times, the squad spirit under new boss Gordon Strachan is good.
He added: "I think it is very easy under these circumstances to let divisions come between us, but there is definitely not that here and we are sticking together as a group and we will come out fighting."