Robert Lewandowksi fired the hosts into the lead on 36 minutes but within two minutes Real were level as Cristiano Ronaldo lobbed in his fifth Champions League goal of the season.
However, Dortmund were the stronger side after half-time and deservedly got the winner midway through the half as Iker Casillas' weak punch was seized upon by Schmelzer, who sealed a famous night for Jurgen Klopp's men.
Real have now only won once in 24 games in Germany but it was nearly a familiar face that undid the hosts early on as Ronaldo burst down the left after just six minutes and his low cross should have been put away by Mesut Ozil, but the German international prodded wide.
Casillas was then called into action for the first time on 12 minutes by a low effort from Sebastian Kehl, before Real's other Germany international, Sami Khedira, was replaced by Luka Modric after pulling up with a muscle injury.
The change did not seem to unduly unsettle the visitors, though, as they continued to have the better of the play, but both Angel Di Maria and Ronaldo fired wastefully off target when presented with good shooting opportunities.
Kehl forced Casillas into an even better save with another long-range effort just before the half-hour mark from Mario Gotze's cutback, but the Bundesliga champions did go in front eight minutes later when Kehl pounced on a wayward pass by Pepe to release Lewandowski and he made no mistake with an emphatic finish inside Casillas' left-hand post.
The lead only lasted seconds, though, as Ozil's long ball forward was not cut out by Lukasz Pisczek and Ronaldo comfortably lofted the ball over the advancing Roman Weidenfeller who should probably have stayed on his line.
The home side reacted brightly to the equaliser at the start of the second period and Casillas had to be sharp to turn behind efforts from Gotze and Marco Reus, however, as is so often the case, Real were enjoying the space they were being allowed to counter-attack into and they should have gone in front but Di Maria wasted clean-through chance.
Having saved his side on a few occasions, though, it was Casillas who was at fault as Dortmund retook the lead with 26 minutes to go as he could only parry a cross from the right as far as Schmelzer on the edge of the area and his volley into the ground made its way through a crowd of bodies and into the bottom corner.
Things could have been even worse for the Spanish champions 12 minutes later as Gotze had strong claims for a penalty waved away after appearing to have been pushed by Pepe, but, despite the introduction of Gonzalo Higuain from the bench, Real rarely threatened an equaliser, with their best chance coming a minute from time when Michael Essien headed wide from a narrow angle.