In the early game, Manchester City overcame a spirited Hull City 2-0 at the Etihad Stadium thanks to second-half goals from substitute Alvaro Negredo and Yaya Toure.
In a lively start, Hull should have taken the lead after seven minutes when Sone Aluko burst clean through, but he scuffed his shot wide and then Danny Graham had a goal disallowed for offside.
Manuel Pellegrini brought on Negredo for Edin Dzeko at half-time and the decision was rewarded on 65 minutes when the summer signing headed home Pablo Zabaleta's deep cross.
Hull refused to give up, but the game was put beyond their reach in the final minute when Toure curled a sumptuous free-kick over the wall and in off the underside of the bar.
After the game, Manchester City boss Manuel Pellegrini confirmed his desire to add a defender to his squad before Monday night's transfer deadline.
West Ham's unbeaten start to the season came to an end after Jermaine Pennant's late free-kick earned Stoke a deserved 1-0 win at Upton Park.
Just as it looked as though the Potters would not make their dominance count, they struck decisively with eight minutes to go when Pennant whipped a 25-yard shot over the wall and past the flailing hand of Jussi Jaaskelainen.
Hammers boss Sam Allardyce was upset with the lack of attacking threat his side posed but admitted he is unlikely to strengthen before the transfer deadline.
Stoke boss Mark Hughes, meanwhile, hailed a "perfect away performance" from his side and revealed he is "still in the mix" for more signings.
Newcastle, meanwhile, recorded their first win, and first goal, of the season after seeing off an inconsistent Fulham side 1-0 at St James' Park.
The Magpies had plenty of chances throughout the game, with Yoan Gouffran wasting the best when he glanced a header wide from close range in the second half.
However, he was spared by Hatem Ben Arfa four minutes from time after the winger cut into the box from the right and hammered a curling shot beyond the rooted David Stockdale.
Norwich are also up and running for the season after summer signing Nathan Redmond's goal secured a 1-0 triumph over Southampton at Carrow Road.
The breakthrough finally arrived midway through the second half when Leroy Fer's cross-field pass found Redmond, who cut in from the left and fired a low shot into the bottom corner.
In the other 3pm kick-off, Manchester United targets Leighton Baines and Marouane Fellaini both started as Everton were held to a 0-0 draw at Cardiff City - with Roberto Martinez yet to record his first league win as manager.
Everton could easily have been awarded a penalty before half-time when Baines appeared to be brought down by Gary Medel, but the referee waved play on.
And Nikica Jelavic was then denied by a superb save from David Marshall, who had to change direction to keep out the Everton striker's deflected header.
In the late game, Dwight Gayle, Danny Gabbidon and Stuart O'Keefe grabbed their first Premier League goals as Crystal Palace opened their points account with a 3-1 win over Sunderland.
It took just nine minutes for the deadlock to be broken at Selhurst Park, with some questionable defending from a set-piece allowing Gabbidon to bundle Palace ahead.
Steven Fletcher made a timely return to the Black Cats set-up as he stepped off the bench at the interval and levelled matters on 64 minutes with a trademark header from a Jack Colback cross.
Palace were not to be denied, though, and went back in front on 79 minutes when record-signing Gayle blasted home from the penalty spot after being upended by John O'Shea - who saw red for a reckless challenge inside the box.
A first win back in the big time was wrapped up in spectacular style in stoppage-time when substitute O'Keefe rifled a 20-yard drive in off the underneath of the crossbar - to leave manager Ian Holloway dancing on the sidelines.
Ian Holloway was delighted with Palace's showing and confirmed the signings of Jack Hunt and Jimmy Kebe from Huddersfield and Reading respectively.
Sunderland boss Paolo Di Canio, meanwhile, rued "pathetic" goals conceded and said the performance proved the need for further signings.