Gary Johnson's men were made to work hard for their three points as Millwall's new-look strike force of Andy Keogh and the returning Steve Morison, along with winger James Henry, all had chances to win the game.
But Yeovil stood firm and nicked the result at the death through midfielder Upson.
The visitors had their goalkeeper Marek Stech to thank for making a number of important saves to keep their side in the game before Upson chanced his luck with just three minutes remaining.
It was Millwall, under new manager Steve Lomas, who had the first real chance of the match when Shane Lowry sent a powerful cross in for Morison but the on-loan Leeds striker could not keep his free header on target.
Minutes later, it was Keogh who headed wide of Stench's goal with a glancing effort.
Yeovil's first effort came through James Hayter, who tried to lob David Forde only to see his effort go just wide.
They had an even better chance after 15 minutes when debutant Dan Seabourne's downward header was well saved by Forde on the line.
For all of their good work going forward, Yeovil struggled to find the openings in front of goal.
Probably the best chance of the first half fell to Henry. The midfielder played a neat one-two with Morison and saw his effort smothered by Stech when he looked favourite to score.
Henry then turned provider as he got to the byline to whip in a cross which Keogh snatched at and sent wide - with Morison better positioned behind him.
It was Henry who continued to cause problems in the second half, watching his effort tipped over the bar by Stech as Millwall looked the brighter after the restart.
Henry was again at the heart of the action soon after as he missed an open goal. Morison charged down Byron Webster's misplaced back pass and the ball fell to the Millwall winger 25 yards from goal, but he could not keep his shot on target despite Stech being stranded.
Lions boss Lomas introduced summer signing Nicky Bailey after the break and the former Middlesbrough man tried to create chances for Morison, but with little luck.
As the game looked to be heading for a goalless draw, Yeovil pushed forward and grabbed the winner.
Millwall failed to deal with a corner and the ball fell to Upson on the edge of the area. He took the ball down well and fired it into the bottom corner past a helpless Forde.
The home side looked stunned at the late goal and little time to recover as Yeovil hung on for their first win of the season.