Hamilton netted either side of the break, with Dennis Wyness also on target to cancel out Neil Janczyk's leveller.
That was enough to see the Scottish Premier League side through to the fifth round and make for an unwanted debut in charge of Brechin for Hamilton's old Dundee coach.
Duffy, who elected to watch his first match in charge from the Glebe Park stands, was dealt a blow in the build-up as first-choice goalkeeper Craig Nelson pulled out due to a close family bereavement.
He was forced to field the relatively inexperienced Chris McCluskey as Brechin tried to conjure up the first genuine upset of the fourth round.
However, it was the Paisley side who looked more comfortable in the opening exchanges with two early chances to take the lead.
Firstly, Argentinian wing-back Franco Miranda curled one of his trademark free-kicks wide from 20 yards after four minutes. Then, from a similar range, Stephen McGinn went even closer.
Gus MacPherson's side did not have it all their own way, though, and a poorly struck free-kick by Hugh Murray allowed Kevin Byers a shot on goal from 20 yards. Again, Byers could not test Saints keeper Mark Howard and the chance was lost.
Byers was left to rue that miss as the visitors took the lead from after 24 minutes.
Englishman Andy Dorman beat John Ward for pace on the corner of the box before falling to the ground under the defender's challenge.
Referee Stephen Finnie wasted no time in pointing to the spot and Hamilton sent Chris McCluskey the wrong way.
That goal gave Saints the impetus and they had a series of chances to make their lead more comfortable after the break.
Wyness had the best of those opportunities when, in the 34th minute, the unmarked striker fired a Hamilton flick-on into the side-netting to leave his side with just a single goal advantage at half-time.
Brechin wasted no time in getting themselves right back into the game with a goal 46 seconds after the re-start.
There seemed no immediate danger when Gary Twigg headed the ball hopefully across goal but the onrushing Janczyk was alert enough to volley superbly past Howard from 15 yards.
They then could have taken a shock lead as Darren Smith found space inside the penalty box at a Kevin Byers corner only to shoot over.
However, St Mirren restored their advantage after 56 minutes when home keeper McCluskey dropped a Miranda cross straight into the path of Wyness and the ex-Hearts star slotted in from 10 yards.
The game was now slipping from Brechin's grasp and worse was to follow as their keeper made yet another costly blunder to gift St Mirren their third goal after 71 minutes.
McCluskey raced off his line to try and connect with a hopeful Jack Ross clearance, only to collide with team-mate Ward and allow Hamilton to nip in and drill the ball into the empty net.
It was a cruel blow to McCluskey and to a Brechin side who had looked at times as though they could trouble their more fancied visitors.
However, with Janczyk firing wide a late free-kick that would have been little more than a consolation for Brechin, it was St Mirren who were in the fifth round with a meeting at either Inverurie Locos or Motherwell next month.