The hosts went ahead in the 35th minute when Stewart Malcolm hammered a clearance from the byline off the back of fellow Red Lichties defender Alex Keddie and into the net.
The SPL champions created and missed a host of chances while also being foiled by Arbroath goalkeeper Scott Morrison, but with four minutes remaining Steven Doris shocked the home side when his deflected drive beat Lukasz Zaluska in the Celtic goal.
Aberdeen and Motherwell will also have to do battle again as Niall McGinn continued his predatory form this season as he headed home a last-minute equaliser to earn a 1-1 draw at Pittodrie.
It had for the most part been an even match between the two SPL sides although Motherwell looked to be heading through thanks to Jamie Murphy's superb strike with 10 minutes remaining before McGinn levelled the scores.
Richie Foran scored in the fifth minute of added time as Inverness snatched a draw in a 3-3 thriller against Ross County.
Billy McKay had put the visitors ahead at the Global Energy Stadium on the half hour only for Rocco Quinn to equalise four minutes into the second half.
Foran put Caley back in front with 16 minutes to go back County were level again when Iain Vigurs netted with four minutes left on the clock.
Richard Brittain put the home side ahead for the first time in the second minute of added time but Foran ensured the spoils were shared.
St Mirren progressed with a comfortable 2-0 victory over Brechin City thanks to goals from Kenny McLean with a well struck free-kick just after the half-hour mark and Jon Robertson four minutes before half-time.
Johnny Russell's hat-trick and a brace from Jon Daly secured a routine 5-0 win for Dundee United at Stranraer, while Kilmarnock beat Queen of the South 2-1 thanks to goals from Cillian Sheridan and Borja Perez, with Nicky Clark on target for the visitors.
Forfar progressed to the next round after a 2-1 win over Ayr, while Dundee were 2-0 winners at Livingston. Dunfermline won 1-0 at Partick Thistle, Raith Rovers beat Deveronvale 2-1, Falkirk were 1-0 winners at Stenhousemuir, while Turriff United drew 1-1 against Morton.