Jose Mourinho's men appeared to have their title challenge back on track after three successive wins but their momentum was halted on Saturday when Oussama Assaidi netted the winner as Stoke registered their first league victory over Chelsea since 1975.
Chelsea seized control of the game early as German striker Andre Schurrle put the visitors in front nine minutes in before failing to capitalise on numerous chances.
Goals from Peter Crouch and Stephen Ireland either side of the break turned the game on its head but Schurrle soon levelled for Chelsea.
However, there was to be another twist as Stoke substitute Assaidi raced down the left in the 90th minute before finding the top corner, leaving Chelsea exploited at the back; something that left defender Cahill very disappointed.
"It's massively disappointing because it's so tight at the minute that you've got to keep chalking up results," he said after the 3-2 reverse.
"A game which we felt we could get points from, we've lost, which is very disappointing.
"When it's at 2-2, it swings either way and we've got to push for the win because we need to try and get the three points, but it opens you up a little bit and that's what happens."
Twice in a week Chelsea have conceded three goals following their 4-3 win over Sunderland in midweek.
They have now conceded more goals in the league than in the whole of Mourinho's first season in charge in 2004-05, when they won the title, and England centre-back Cahill is disturbed by the current situation.
"Losing games is a concern," he added. "We've kept a lot of clean sheets this season, but in the last two games, we've let too many goals in. That's unlike us and it's down to individual mistakes.
"We talked about it after Sunderland and there were unfortunate incidents at the set-pieces. We've had a look at it and they were not major, major mistakes. Maybe it's just a bit of bad luck but obviously, away from home, we've got to keep it tight.
"I feel we've done that most of the season, but this result was poor for us."
Chelsea have the opportunity to get back to winning ways in the Champions League on Wednesday, knowing victory over Steaua Bucharest would guarantee them top spot in Group E.