McCarthy somehow held onto Alexander's low drive from close range in a game of precious few clear-cut chances.
The home side, with winger David Martin prominent, pumped a succession of balls into the Yeovil box in the opening half but found the visiting defenders up to the task.
The closest the Lions came to a goal was just after the half-hour mark, when Steve Morison's looping header from James Henry's corner deceived goalkeeper McCarthy, only to strike the top of the crossbar.
In fact the main talking point came when Yeovil's Sam Williams caught Lions keeper David Forde with a flailing arm, leading to a shoving contest that saw both men booked.
The game improved after the break and within the space of two minutes both Shaun MacDonald and JP Kalala clattered the woodwork with long-range drives for the visitors.
Substitute Alexander added more bite to the Millwall attack, but he was also unable to direct two close-range headers on target.