Dean Shiels and Kal Naismith were also on the scoresheet to help ensure Gers' safe passage into the fifth round of the competition.
Both clubs would have welcomed the opportunity simply to play this match after last week's Third Division encounter was postponed when it emerged Elgin had sold too many tickets for the visit of Ally McCoist's men.
The match was less than two minutes old when Rangers had the ball in the back of the net through Kyle. But he was denied the opener by the offside flag and the decision looked like it may have been harsh on the striker.
Rangers continued to threaten and the lively Barrie McKay surged into the box from the left before teeing up the shot for Kyle but he connected awkwardly in front of goal and the chance was lost.
At the other end, Neil Alexander was called into action to block an angled drive from Craig Gunn.
The home side still looked the more likely to find the back of the net and Shiels stung the palms of goalkeeper John Gibson with a powerful shot. Gibson was called into action again moments later, this time to tip wide an impressive 30-yard effort from Ian Black.
With half-time approaching and the match still deadlocked, Lee Wallace burst past Jamie Duff down the left flank before cutting back for Kyle but he fired wide.
Shiels finally had the net bulging two minutes before the break with a shot from a tight angle that fell across the goalkeeper and nestled in the far bottom corner.
Gibson was picking the ball out of the net again after 69 minutes when Kyle rose above everyone in the box to connect with a McKay corner and nod past the grasp of Gibson.
Only a great save from Alexander prevented Sean Crighton from pulling a goal back for Elgin, before David Templeton was thrown into the action with just under 20 minutes to go.
McKay was denied by the woodwork when his deflected effort crashed off the post, before substitute Naismith put the gloss on the win when he slotted past Gibson with 85 minutes on the clock, seconds after replacing Kyle.