Vale goalkeeper Chris Neal saved Reuben Reid's 80th-minute penalty following Chris Lines' foul on home skipper Conor Hourihane, for which Lines was dismissed, but the visitors held on.
Argyle got off to a flying start thanks to Argentinian winger Andres Gurrieri, who headed them ahead after 90 seconds. He stooped to beat Neal to Lewis Alessandra's pacy cross from the left.
Vale responded in the 29th minute when targetman Jordan Hugill scored with a superb low drive into the corner from 18 yards following a one-two with Louis Dodds on the edge of the box.
Argyle's response was swift as Hourihane drove forward on a defence-splitting run from the halfway line before smashing a 25-yard shot on the run into the bottom corner after 35 minutes.
Vale levelled when substitute Ben Williamson headed home a cross from Myrie-Williams in the 63rd minute before the latter cracked in the winner.
Micky Adams' side will now host Brighton in the fourth round.