Gary Hooper's opener just before half-time on Sunday was cancelled out three minutes into the second half when Hoops' stopper Glenn Loovens put through his own net as he tried to clear Kyle Lafferty's header.
Miller then fired the visitors ahead with a volley in the 55th minute before netting from the spot 10 minutes later after Old Firm debut referee Willie Collum had pointed to the spot when Daniel Majstorovic had fouled Light Blues' substitute Kirk Broadfoot inside the box.
It was statement of intent from the champions, who extended their winning run in the league to nine games while at the same time ending their bitter rivals' winning run in the SPL this season.
While there was no difference between the teams in points going in the game there was a huge difference in their respective experience of Old Firm games.
Celtic, with the same side which beat Dundee United at Tannadice last week, had six players making their debut in the fixture.
Rangers manager Walter Smith, who started the game watching from the stand, made two changes to the side that drew 1-1 with Valencia at Ibrox in the Champions League in midweek. Midfielder Lee McCulloch returned after injury and Lafferty also came in as Richard Foster and Vladimir Weiss dropped out.
All eyes were on referee Collum and it took him only 70 seconds to produce a yellow card for Anthony Stokes, for a crunching challenge on Ibrox defender Sasa Papac.
Then, after Steven Whittaker had powered a 20-yard shot off the base of the post, and Steven Naismith had his drive from the rebound blocked, McCulloch was booked for a foul on Hoops' left back Emilio Izaguirre.
In the midst of the initial madness, with bodies falling to the ground all over the pitch, Rangers midfielder Steven Davis tested Celtic goalkeeper Fraser Foster with an effort from 20 yards.
Parkhead skipper Shaun Maloney went off with concussion in the 14th minute after a clash of heads with Naismith, to be replaced with another Old Firm debutant, Efrain Juarez.
On the half-hour mark Davis had another shot, this time on the turn from 16 yards and again Forster dealt with it easily before Izaguirre fired over from the distance at the other end.
In the 33rd minute, when McCulloch brought Georgios Samaras tumbling to the ground 30 yards out, the Ibrox bench and supporters held their breath lest Collum reached for another yellow card for the midfielder. However, the referee showed leniency and Ibrox keeper Allan McGregor made a decent save from Ki Sung-yong's free-kick.
Just before the interval Ki's corner from the left came off the head of Whittaker and headed for Hooper. The former Scunthorpe striker quickly brought the ball under control and hammered it past Davis, who was guarding the far post.
Broadfoot replaced Papac at the break and within three minutes the visitors drew with the help of Lafferty, who had been sandwiched to win a free-kick.
Davis floated the ball in and his Northern Ireland team-mate headed towards goal, with Loovens only helping it in as he tried to clear.
The home side were the architects of their own downfall again in the 55th minute when Miller dramatically put Rangers ahead.
Forster made a mess of a routine clearance and Naismith quickly tried to put Maurice Edu through with Miller looking to be in an offside position.
However, when the ball broke to the former Hoops' striker 16 yards out, he buried his volley low past Forster.
More drama arrived in the 65th minute when Collum awarded a penalty to the Govan club after Majstorovic brought down Broadfoot in the box, although contact seemed minimal.
After the referee brushed aside Celtic complaints Miller drove the spot-kick in off the post.
The Parkhead men fought to get a foothold back in the match and McGregor made good saves from Ki and Joe Ledley - but Rangers were happy to see the game out.