The three previous Edinburgh derbies in the league this season had ended in draws and this game also appeared to be heading for a draw after Leigh Griffiths had cancelled out Darren Barr's opener.
However, Caldwell, 19, who was making just his second start, struck at the death to send Hibs above the Jambos into ninth place in the table.
After an early battle for midfield supremacy it was Hearts who twice went close to taking the lead in the 11th minute when both Barr and Michael Ngoo saw close-range headers blocked on the line by Lewis Stevenson and Alan Maybury respectively after Ben Williams had spilled Kevin McHattie's corner.
Teenage winger Alex Harris was looking lively for Hibs and he tested Hearts goalkeeper Jamie MacDonald with a low, deflected shot from the edge of the area.
The visitors had appeals for a penalty waved away after Kevin Thomson had tangled with Ryan Stevenson and Caldwell's deflected effort was then saved by MacDonald.
Jamie Walker curled a shot wide as Hearts pressed before the interval and they earned their reward in the 45th minute when Barr struck.
The defender scored his first goal since last year's Scottish Cup final when he hooked home off the ground from close range after Ngoo's header from McHattie's corner had been blocked on the line by Tom Taiwo.
Hibernian were level in stunning fashion just three minutes after the interval, though, when Griffiths' 30-yard free-kick curled into the top right-hand corner of the net with MacDonald grasping at thin air, giving the striker his 100th goal in Scottish football.
Griffiths was inches away from adding a second moments later when he just failed to convert Harris' centre, but Hearts then enjoyed a period of pressure.
Jamie Hamill's free-kick was swatted away by Williams and Walker's volley bobbled just wide before Jordon Forster cleared Ryan Stevenson's centre.
Hibs held firm, however, and they snatched the three points in the 90th minute when sub Scott Robertson cut in from the left before passing to the unmarked Caldwell who took his time before curling home a right-foot shot to the delight of the visiting fans behind the goal.