Stoke City were also 1-0 winners after getting the better of West Ham United, while Everton drew for the third successive game, this time against Cardiff, and Manchester City beat Hull City at home.
Man City overcame a spirited Hull side 2-0 at the Etihad Stadium thanks to second-half goals from substitute Alvaro Negredo and Yaya Toure.
In a lively start, Hull should have taken the lead after seven minutes when Sone Aluko burst clean through, but he scuffed his shot wide and then Danny Graham had a goal disallowed for offside.
Manuel Pellegrini brought on Negredo for Edin Dzeko at half-time and the decision was rewarded on 65 minutes when the summer signing headed home Pablo Zabaleta's deep cross.
Hull refused to give up, but the game was put beyond their reach in the final minute when Toure curled a sumptuous free-kick over the wall and in off the underside of the bar.
Manchester United targets Leighton Baines and Marouane Fellaini both started as Everton were held to a 0-0 draw at Cardiff City as Roberto Martinez awaits his first league win as manager.
Everton could easily have been awarded a penalty before half-time when Baines appeared to be brought down by Gary Medel, but the referee waved play on.
And Nikica Jelavic was then denied by a superb save from David Marshall, who had to change direction to keep out the Everton striker's deflected header.
Newcastle recorded their first win, and first goal, of the season after seeing off an inconsistent Fulham side 1-0 at St James' Park.
The Magpies had plenty of chances throughout the game, with Yoan Gouffran wasting the best when he glanced a header wide from close range in the second half.
However, he was spared by Hatem Ben Arfa four minutes from time after the winger cut into the box from the right and hammered a curling shot beyond the rooted David Stockdale.
Norwich are up and running for the season after summer signing Nathan Redmond's goal secured a 1-0 triumph over Southampton at Carrow Road.
The breakthrough finally arrived midway through the second half when Leroy Fer's cross-field pass found Redmond, who cut in from the left and fired a low shot into the bottom corner.
West Ham's unbeaten start to the season came to an end after Jermaine Pennant's late free-kick earned Stoke a 1-0 win at Upton Park.
The Potters struck decisively with eight minutes to go when Pennant whipped a 25-yard shot over the wall and past the flailing hand of Jussi Jaaskelainen.
Crystal Palace claimed their first points of the season in Saturday's evening kick-off, with Ian Holloway's side running out 3-1 winners over 10-man Sunderland.
Danny Gabbidon knew little about his early opener, as an effort which deflected into the net off his shins saw him become the first of three Palace players to grab their first Premier League goals.
The Eagles' record-signing Dwight Gayle was next to achieve that feat, but only after Steven Fletcher had powered home a trademark header on his return from injury to level matters and referee Lee Probert had pointed to the spot and dismissed John O'Shea for a reckless challenge inside the box.
Victory was then wrapped up in stoppage-time when substitute Stuart O'Keefe rifled home from 20 yards out, with a strike which crashed in off the underside of the crossbar leaving manager Ian Holloway dancing on the sidelines.