The Scot is a proven performer at Championship level, having spent two years at Cardiff prior to making a move to St Andrew's.
He agreed a switch to Birmingham shortly before Alex McLeish left the Blues for Aston Villa.
McLeish is now the man in charge at Forest, and he had been hoping to snap Burke up for a second time.
Forest are in need of reinforcements during the winter window, with it vital that they add strength in depth to their squad if they are to mount a promotion push.
Burke's experience would be a useful asset to bring on board, but Birmingham are reluctant to discuss the current terms on offer.
Lee Clark is, however, under pressure to sell during the winter window and it could be that Forest return with an improved bid before the month is out.