City got off to an explosive start as they took the lead inside the first minute when Cyrus Christie crossed from the right and Leon Clarke poked home from close range.
Gillingham came back and were denied an equaliser when Joe Murphy saved from Amine Linganzi just five minutes later.
But former Coventry striker Cody McDonald returned to haunt his old club when he latched onto a loose ball in the box and fired into the net from 12 yards in the ninth minute.
The Sky Blues had several chances to regain the lead before the break but found Stuart Nelson in good form - denying Clarke, Franck Moussa and Carl Baker with strong saves.
The home side were on top in the second half and both Clarke and Baker struck the woodwork but just when it looked as if City would have to settle for a point, Moussa turned into the net from close range in the 82nd minute.