Youngster Barrie McKay opened his Rangers account with what should have been a goal to savour, before second half goals from Rory McAllister and Scott McLaughlin saw the home side surge into the lead.
But Little grabbed a late equaliser and a share of the points for Ally McCoist's men.
The match at Balmoor was the first ever league fixture played by the Glasgow giants outside the top flight of Scottish football.
The Ibrox side were forced into the bottom tier after the club was consigned to liquidation in the summer and lost their place in the SPL following a vote of the 12 clubs.
Following such a turbulent period in their 140-year history, the meeting with the Blue Toon was seen as the first step on the road to recovery.
Rangers named an unchanged side from the one who defeated East Fife in the Scottish Communities League Cup.
New recruits Ian Black and Dean Shiels retained their places in the starting line-up following the midweek 4-0 win at Ibrox and Francisco Sandaza and Kyle were again named on the bench.
But Emilson Cribari - who was still awaiting international clearance yesterday and who had made the long trip north - was not in the squad.
Rangers found their hosts difficult to break down early on and it was the home side who carved out the first real chance of the game when McKay was called into action to clear a McAllister header.
At the other end, Shiels had a go with a shot from the edge of the area but his effort was comfortably smothered by goalkeeper Paul Jarvie.
Rangers were threatening again and, this time, Andy Little's shot cannoned off the body of Steven Noble and out for a corner that the visitors failed to capitalise on.
The Ibrox side were able to breathe a sigh of relief when they opened the scoring after 26 minutes and the honours went to McKay.
A long ball from Lewis McLeod was knocked on by Lee McCulloch for the 17-year-old, who kept his cool to drive a low shot into the back of the net.
McAllister then had a great chance to draw level for Peterhead with a shot from six yards that he dragged across the face of goal and wide of the upright.
McKay could have added to his own tally before the break but the teenager was also off target and Rangers had to settle for a slim advantage at half-time.
Rangers goalkeeper Neil Alexander was promoted to number one following the departure of Allan McGregor.
And he produced an impressive one-handed save after the restart to prevent a lovely swerving effort from McAllister from finding the top corner.
But the forward had been a menace throughout the match and was finally rewarded with a goal after 64 minutes when he did well to hold off Carlos Bocanegra before lashing into the bottom corner.
Rangers threw Sandaza into the action for McKay but it was Bocanegra who could have restored the lead when he met Black's free-kick only to nod wide from close range.
Former St Johnstone striker Sandaza then tried to chip Jarvie but lofted into the goalkeeper's arms instead, before Shiels was denied by a save at the near post.
But Rangers were left reeling when Peterhead had the net bulging again with nine minutes to go when Ryan Strachan's corner was cleared into the path of McLaughlin and he rifled past a stunned Alexander.
The Light Blues were back on level terms with a minute to go when substitute Kyle rose above everyone else in the box to meet a Shiels corner and although his header came back off the bar Little was on hand to score.