After 90 minutes had failed to produce a goal at Wembley, Palace eventually scored the winner their second-half dominance deserved just before the halfway point in extra-time, when Manchester United-bound Wilfried Zaha earned a spot-kick, which was ruthlessly dispatched by substitute Phillips.
The victory will see Ian Holloway's side return to the top flight after an eight-year absence and bank an estimated £120million. Gianfranco Zola's Watford, who were never allowed to get into their stride, must try again next season after narrowly missing out on automatic promotion on the final day of the season.
The two previous Championship meetings between the sides produced nine goals but the first half at Wembley was a tight and tense affair, with neither goalkeeper tested.
Watford dangerman Matej Vydra was given a sight of goal on 33 minutes when neat interplay around the edge of the box saw the ball break for the Championship Player of the Year. But as he looked to bend a shot with his right instep, Damien Delaney threw himself across to get in a splendid block.
The challenge left Vydra with an ankle knock and the Czech forward was unable to continue after half-time, with Spaniard Alex Geijo coming off the bench to start the second half.
The game's first real opportunity came just before the hour mark, when Aaron Wilbraham was played in behind the Watford defence. The Palace forward, though, was eventually reined in by the chasing centre-halves, who were allowed to get goal-side and block when Wilbraham tried to cut inside and shoot.
Manuel Almunia was finally tested in the 65th minute, when the former Arsenal goalkeeper made a fine block to keep out substitute Stuart O'Keefe's firm drive after good hold-up play by Zaha.
From the resulting corner, Almunia was forced to get down low again when no one in the Watford defence took responsibility for clearing a bouncing ball, gifting an opportunity to Wilbraham. The goalkeeper, though, sprang from his line to smother the Palace striker's shot.
With Palace in the ascendancy, Holloway sent on Phillips for the last quarter, and the veteran forward was soon involved in the 72nd minute. Zaha's deep cross was nodded back by Phillips to Owen Garvan, but his shot drifted wide of the far post.
Garvan did find the target with eight minutes remaining with a volley after a brilliant touch and spin in the box, but Almunia was well positioned to make the catch, just as he was two minutes later when Wilbraham created a yard of space but his shot was blocked by the keeper's left foot.
Palace tested Almunia once more in added time, when Mile Jedinak headed a corner towards goal but the Watford stopper sprang to his left to make a diving catch.
After remaining untested during normal time, Palace keeper Julian Speroni made a wonderful save early in extra-time when he dived to his left at the feet of Troy Deeney to deny the Watford striker a tap in from two yards. Speroni then stood firm to hold Cristian Battocchio's swerving drive from 25 yards.
The breakthrough finally arrived in the 105th minute. Zaha's trickery fooled Marco Cassetti, who was tempted into a challenge that sent the Palace winger crashing to the Wembley turf. Phillips took the spot-kick and smashed the ball into the top corner beyond the dive of Almunia.
Watford pressed late on and Palace were thankful for two interventions by defender Joel Ward, the second of which was a header off the line to keep out Fernando Forestieri's effort.