The home side opened the scoring after just four minutes through an own goal from Anestis Argyriou.
But the Irn-Bru Third Division leaders levelled thanks to a Dean Shiels penalty, before Andy Little netted his fifth goal against the English side this season.
Rangers had been far from impressive on their last visit to Shielfield, scraping an unconvincing 1-1 draw at the start of the season.
And they endured the worst possible start to this encounter when they found themselves trailing after just four minutes.
The visitors failed to deal with a Neil Janczyk corner into the box and the ball hit off Argyriou - under pressure from Darren Lavery - and into the back of his own net.
Rangers were back on level terms five minutes later when referee Calum Murray pointed to the spot after Shiels went to ground under a Dougie Brydon challenge.
The penalty award may have been harsh on the home side but the Light Blues restored parity when Shiels dusted himself off to rifle straight down the middle from 12 yards.
With half-time approaching, the home side squandered another chance when Easton latched onto a throw-in and fizzed across the face of goal from the tightest of angles.
Instead, it was Rangers who edged in front in injury time. Lee Wallace supplied the cross and Little sent a lovely left-foot volley beyond the grasp of Marc McCallum.
Faure entered the pitch on 53 minutes and the Frenchman opened his account for Rangers after 66. A David Templeton corner into the box fell for Naismith, his header was blocked, and Faure pounced to rifle home the rebound.
Substitute Dean Carse saw a decent effort blocked by Faure and Kenny O'Brien also passed up a decent chance to haul the Wee Rangers back into the game as the Glasgow side held on for the win.