The Gunners had much the better of the first half but could not make their dominance count and Joleon Lescott headed the hosts in front from a 40th-minute corner.
That looked like being the way it would stay as Arsenal struggled to find a way to goal, but Koscielny proved the unlikely hero with an unstoppable shot eight minutes from time.
Both sides wore black armbands to remember Greater Manchester Police officers Fiona Bone and Nicola Hughes, who were killed on Tuesday, and all sides of the ground stood for a period of applause before kick-off.
City's defence of their Premier League title had consisted of two scrappy wins and two draws, while they suffered a last-gasp defeat by Real Madrid in the Champions League on Tuesday.
They were at least boosted by the return of Sergio Aguero for the first time since the opening weekend of the season after a knee injury, and he had the first real chance of the game in the 14th minute with a curling shot that was pushed away by Vito Mannone.
It was Arsenal who were looking the more dangerous, though, cutting swathes through City's midfield, and had it not been for a truly awful touch from Gervinho as he ran on to Aaron Ramsey's through ball, they may well have been ahead.
The Gunners were taking on their opponents with real pace and in the 24th minute Carl Jenkinson did well down the right and pulled the ball back to Lukas Podolski, who skied his shot well over the bar.
Gervinho then fired just wide with a shot that rippled the side-netting and drew cheers from the section of Arsenal fans down the other end who thought it had sneaked in.
Still they had not really tested Joe Hart, though, and they were made to pay in the 40th minute when Lescott, who had been surprisingly left out against Madrid, headed in David Silva's corner.
Mannone, playing in place of the injured Wojciech Szczesny, certainly did not cover himself in glory, attempting to plough through a ruck of defenders and getting nowhere near the ball.
City could even have been two ahead at the break but Mannone partially redeemed himself with a fingertip save from Edin Dzeko's curling shot.
City boss Roberto Mancini addressed his midfield issues by bringing on Jack Rodwell for the disappointing Scott Sinclair at the start of the second half, and immediately there was more dynamism about the home side.
Aguero fired over at the back post and was then screaming for a penalty when he thought he had been tripped by Koscielny.
The Argentinian stayed on his feet and managed to get his shot away, which was blocked by Kieran Gibbs, and that probably counted against him as referee Mike Dean waved away the claims.
Arsenal were still passing the ball around neatly without really looking like scoring, and Gervinho's radar was off target again as he blasted high and wide in the 66th minute.
Dzeko was then replaced by Carlos Tevez while Arsenal boss Arsene Wenger made a double change, Theo Walcott and Olivier Giroud coming on for Lukas Podolski and Abou Diaby.
Arsenal were pressing hard but City almost killed the game off on the break in the 80th minute, Yaya Toure playing in Aguero for a shot that was smartly stopped by Mannone.
That proved to be a crucial moment as Arsenal went up the other end and Santi Cazorla drew the first real save out of Hart with a fine shot from the edge of the box.
The Spaniard took the resulting corner and Lescott could only divert the ball to the feet of Koscielny, who smashed the ball into the net.
It was no more than Arsenal deserved but they had Mannone to thank for keeping them level moments later as he saved Vincent Kompany's acrobatic overhead kick before Aguero shot inches wide.
The Gunners had the final chance but again Gervinho's finish did not match his approach play, the Ivorian firing well over from 15 yards out.