The U's came into the game in second place, having won four of their five away games, but they keep struggling to deliver in front of their own fans and have won just once at the Kassam Stadium this season.
Wilder said: "It was individual errors that cost us. Southend didn't really have to work that hard for their goals.
"The first goal we gave the ball away with a poor clearance and their second goal was a free-kick given away by an individual error and then we defended it really poorly.
"Goals change games. The first goal gave them a massive lift and the second one lifted them even more.
"Who knows if things might have been different had James Constable converted Callum O'Dowda's great ball in?
"The game gets decided in both penalty boxes and we didn't do well enough in both boxes.
"We have dropped down to fourth in the table, which is very disappointing, but we have to take defeats on the chin, just as we take the plaudits when we win.
"It's a tight competition, there are no mug sides. Southend have been a strong team in this division for a couple of years.
"But we need to be better collectively, individually and to produce what we produce through the week on the training ground when we play at home.
"It's so disappointing to give away daft goals, like we have done."
First-half goals from Conor Clifford and centre half Luke Prosser secured the win, Southend's second on the road this season, and they might have scored more in the second half.