The Conference South outfit threatened an unlikely comeback when Reece Prestedge fired home with nine minutes to go but League Two's bottom side held on for just their third win of the season.
The Cobblers nearly had the early breakthrough when Mathias Kouo-Doumbe's header from Chris Hackett's corner was cleared off the line by Johnny Herd.
Town had the best of the early chances, but their two on-loan strikers, Norris and Jacob Blyth, could not beat Joe Wright in the Bishop's Stortford goal.
In the 20th minute Kane Ferdinand saw his effort glance off a post, and minutes later Sean Francis' header for Bishop's Stortford cannoned off the bar at the other end.
Kyle Vassell, whose ten goals in qualifying propelled Bishop's Stortford into the first round, limped off injured in the opening minutes of the second half.
And Northampton's pressure finally paid off on the hour, with Emerton firing in from 25 yards.
Then Norris, on loan from Brentford, doubled the lead eight minutes later with an effort from a similar position.
Prestedge got Bishop's Stortford back into the game with a thumping shot from the edge of the box but they could not find the second to force a replay.