Lewis Macleod returned from illness to claim an early opener for the league leaders, who looked set to hold on for the points and their 12th win in a row.
But the 46,406 crowd were stunned into silence when Alexander fumbled a Stewart Leslie header into the back of his own net to gift Elgin a share of the points.
The Light Blues did not have to wait too long for the opener and were ahead with just nine minutes on the clock.
Lee McCulloch won the ball in the middle of the park before releasing Macleod into the box and he coolly dispatched a low shot past Joe Malin into the back of the net.
The visitors were reduced to 10 men four minutes before half-time when Paul Harkins was shown a straight red card for a foul on Ian Black.
McCulloch should have claimed Rangers' second of the day just before the break when he collected Lee Wallace's cross from the left six yards out but a poor finish allowed Malin to block easily.
Rangers dominated possession after the restart but struggled to carve out much in the way of clear chances, although David Templeton was not too far away with a volley from the edge of the area that fell just over the crossbar.
Barrie McKay was thrown into the action after an hour, replacing goal-scorer Macleod, and almost immediately he called Malin into action with a shot straight at the goalkeeper.
Rangers made another change five minutes later, when Ross Perry was forced out of the action following a clash of heads, and was replaced by fellow defender Sebastien Faure.
Instead, an horrific error from Alexander gifted Elgin the equaliser with three minutes to go.
Moore's free-kick was met by Leslie who directed a header at goal, which initially appeared to be blocked by the goalkeeper before he fumbled the ball back over his goal-line.