The Black Cats struggled to create much during a disappointing first-half showing and entered the interval one-goal down.
They levelled matters through Steven Fletcher, as he marked his return from injury with a trademark header, but then found themselves behind again and down to 10-men after skipper John O'Shea was shown a red card for tripping Dwight Gayle inside the box.
Palace hit a third late on to compound Sunderland's misery and Di Canio admits they have much to work on defensively if they are to recover from a slow start to the campaign which has delivered one point from a possible nine.
He told Sky Sports: "That hurt me. When you concede the goals that we conceded today, I don't think you can be happy.
"There is a job to do because we were on top for a long period of the game. This is why I'm not worried when I talk about football expression, but if we are going to concede goals like we did today then we have to be worried.
"This game was crucial for us, for everybody. It won't decide the outcome of our season, but if we don't start thinking that every ball, every challenge can be crucial, that could be a problem.
"After our goal we were on top, we played well, we had two or three situations where we should do better and then we conceded another two how we did. It's absolutely poor."
Asked if he may need to dip back into the transfer market before Monday's deadline, after a busy summer of recruitment at the Stadium of Light, Di Canio added: "It's obvious. My eyes were open and now everybody can understand even more why I asked for a few players in a few areas of the field.
"But anyway, it's not the time to be talking about adding to my squad. In terms of performance we did well in general, were on top for most parts of the game, but conceded three very, very pathetic goals. That didn't give us the chance to go back to Sunderland with points."