The Eagles' promotion aspirations continued to falter as Frank Nouble netted twice either side of a stunning long-range volley from Aaron Cresswell at Portman Road.
And Holloway, who took charge of Palace last year, has insisted his side must bounce back from the blow.
"I was more than happy up until the first goal," he said. "I thought we were fantastic, looked comfortable and the best we've been for a long while.
"But in a couple of minutes everything changed. It knocked the stuffing out of us and after that it was difficult to say the least.
"But you've got to believe in yourselves when others don't. It's going away from us and we've got to get it back.
"Life's like that. You have to be brave enough to pull things out of the fire. It's going to be a huge test for us, so let's see if we can bounce back.
"Now's the time to stick together and work our socks off, because the harder you work the luckier you get.
"It's not good enough at this moment in time but I saw all sorts of effort and willingness."