Summerfield's scrambled 47th-minute effort in the early kick-off at St Mary's ensured Peter Reid got off to a winning start as Pilgrims manager.
The defeat is an early setback for Saints boss Alan Pardew, who is under pressure to guide the South Coast side back into the Championship after only narrowly avoiding the sack at the end of last season.
Saints dominated possession in the first half but, with last year's top-scorer Rickie Lambert sidelined by injury, lacked any cutting edge.
Argyle took the lead with their first effort on goal two minutes after half time. Bondz N'Gala's cross was headed across goal by Rory Fallon and Steven McLean flicked the ball into the path of Summerfield, who bundled home at the back post.
Substitute Adam Lallana came closest to salvaging a point for the home side moments after coming off the bench when his 25-yard pile-driver was turned behind by Argyle's on-loan Tottenham goalkeeper David Button.
Substitute centre-back Reda Johnson needed to make a timely late block to deny Lee Barnard in the closing stages, but Argyle held on to take three points.