Saints goalkeeper Kelvin Davis denied David Mooney early on, but Southampton took the lead on the half hour through winger Alex Chamberlain, who converted from close range after Rickie Lambert's free-kick had deflected off the wall and into his path.
Things went from bad to worse for Colchester less than two minutes later when leading scorer Mooney was dismissed for pushing Southampton right-back Danny Butterfield.
Us goalkeeper Ben Williams made a fine save to keep out Adam Lallana's powerful header. But Southampton doubled their lead five minutes after half-time through Hammond, who fired in a low shot from the edge of the area after the home side had failed to clear a corner.
Williams foiled Richard Chaplow and Lee Barnard poked wide from close range after getting the better of John White in the area.
And the Us shot-stopper again thwarted Lallana late on as Saints took the points.