Striker Lee Barnard put the Blues ahead in the 14th minute from the penalty spot after Robbie Williams had brought down Anthony Grant. Barnard calmly slotted the low spot-kick into the right-hand side of the goal with goalkeeper Alex Smithies going the wrong way.
The Shrimpers doubled their advantage in the 56th minute when Anthony Kay's pass was intercepted by Alan McCormack who spotted Moussa unmarked on the left and he shot low into the centre of the goal.
Terriers boss Lee Clark replaced his strikers with Jordan Rhodes and Lee Novak coming on for Theo Robinson and Robbie Simpson on the hour - and Novak almost levelled in the 70th minute when his cross was cleared off the line by Simon Francis.
But a minute later the Terriers were back in it when Gary Roberts' deep cross from the left was volleyed home at the far post by Anthony Pilkington.
Steve Mildenhall made a good parry to deny Novak, but Huddersfield equalised in the 79th minute when Pilkington's ball across the face of goal was headed home from six yards by Rhodes. But the Terriers finished the match with 10 men on the pitch after Michael Collins was sent off in the 82nd minute for violent conduct after shoving Simon Francis.