The visitors dominated from the first whistle but the match looked set to remain goalless as Ally McCoist's side failed to take a string of chances.
But referee David Somers pointed to the spot with 19 minutes remaining when Lee Wallace went down under a challenge from Sean Crighton and McCulloch sent Joe Malin the wrong way from 12 yards.
It was as much as Rangers deserved on the balance of play with City struggling to get out of their own half and doing little to threaten Neil Alexander's goal when they did.
The Glasgow side could have been in front as early as the fourth minute had Malin not pulled off a flying one-handed save to keep out Francisco Sandaza's header.
Robbie Crawford drilled onto the outside of the post six minutes later and a good opening 15 minutes was capped when Sandaza went on a jinking run before teeing up Andrew Little to fire wide.
Chances were few and far between for the rest of the half although Daniel Moore curled a free-kick just over the bar in the best opening for the hosts.
It was a similar story after the interval with Gers controlling the match without finding a way through the Elgin defence, Sandaza's desperate dive at a Little cross summing up the match.
The penalty served to breathe some life into the contest as Elgin went in search of an equaliser but there remained an absence of quality.
And the hosts were given a glimmer of hope when Rangers were reduced to 10 with four minutes to play after Ian Black was shown a second yellow for a clumsy challenge on Gavin Morrison.
But the Rangers defence held firm to restore order following their shock home defeat to Annan Athletic last weekend.