Without a point or a goal going into the match at Whaddon Road, Arygle surprised their hosts by racing into a 3-0 half-time lead.
Marvin Morgan opened the scoring in the 17th minute, following up from close range after Neal Trotman's header from Rommy Boco's cross had the crossbar.
Conor Hourihane doubled Plymouth's advantage 10 minutes later with a fine finish from outside the penalty area.
Reuben Reid's pass split open the Cheltenham defence for number three in the 37th minute, allowing Boco to beat Scott Brown with a low shot.
The shell-shocked hosts missed a chance to hit back two minutes into the second half when Jermaine McGlashan blazed over, but they did reduce their deficit in the 72nd minute.
Matt Richards was fouled by Morgan 25 yards from goal and his free-kick deceived Luke McCormick in the Plymouth goal and ended up in the back of the net.
But Argyle held on comfortably for the final 18 minutes to register their first win, while last season's losing play-off semi-finalists Cheltenham have taken only one point from a possible nine.