The Gunners produced a classic smash and grab to beat last year's beaten finalists thanks to a 62nd minute header from the Welshman.
Arsenal had failed to muster a single shot on goal in the first half but nicked the winner with their first real effort on goal as Ramsey bundled the ball over the line after Olivier Giroud flicked on Mesut Ozil's centre.
The Germans - who had the better openings prior to the Arsenal goal - were left to rue fluffing their lines in the first period as the visitors shut out Robert Lewandowski and co for only the second time in 61 home games.
The Pole had an injury-time appeal for a penalty turned down but the visitors held on to record a famous win.
Dortmund had come into the Group F encounter searching for an eighth successive home Champions League win and had never lost against English opposition.
Jurgen Klopp's side, who had won 2-1 in London a fortnight ago, dominated for long spells, but were made to pay for not converting their chances.
It may not have been the flowing display which manager Arsene Wenger would have preferred - but the Barclays Premier League leaders once again answered questions with a display of real character.
With Napoli beating Marseille, Arsenal still have work to do to secure safe passage through to the knockout stages, but will have gained more belief they can last the distance this season.
Dortmund dominated the first half and were left counting the cost of not making the most of that spell in control, where the English side could not muster any attack of note and Henrikh Mkhitaryan missed a great chance for Dortmund when he shot wide from the edge of the penalty area after Arsenal were carved open down the right.
The Germans pressed again at the start of the second half, with Arsenal once more lacking invention going forward.
Gunners goalkeeper Wojciech Szczesny saved well from Jakub Blaszczykowski following the restart, before Ramsey secured what proved to be the winner just after the hour for his 11th goal of the season.
Arsenal could have added to their lead but Ramsey was denied by goalkeeper Roman Weidenfeller while Koscielny saw an effort cleared off the line.
Koscielny also had a strong claim for a penalty when he appeared to be taken out by the Dortmund keeper who came for a centre and got nowhere near the ball.
The home side stormed forward in the closing stages but the Gunners stood firm to clinch a crucial away win.
Goalscorer Ramsey later hailed a classic European away display from the visitors and told Sky Sports: "It was a great achievement from the lads.
"It is always a difficult place to come. We realise that from the last time we came here and drew 1-1.
"We kept our shape really well, we were patient and it was a proper away performance from the lads and we got them on the counter-attack."