City take on Spanish champions Real Madrid at the Etihad Stadium on Wednesday night and must win to stand any chance of progressing to the knock out stages.
City returned to the top of the Premier League on Saturday after beating Aston Villa 5-0 but Kompany knows that their European form must change.
"I guess there's a big contrast between our Champions League performances and what we've done in the league so far," said Kompany.
"We need to make sure our Champions League performances get better. If I knew the answer as to why, we probably wouldn't be where we are. For some freakish reason we still have a chance in the Champions League."
He added: "I like to think in the two games we have left, we can play to qualify but more for pride as well. We are better than what we have shown so far. It's a great opportunity, we still have a chance.
"There are not many games when we are not favourites. This will be one of those games when we can take more positives out of it than negatives."
The Belgian defender also spoke of the importance of the victory over Villa, commenting: "Our European form is what makes the picture a little less clear. But it's good to give the victory over Aston Villa to our fans. It's probably the most comfortable victory we've had this season against a difficult team.
"It's nice to be back on top of the table but there are so many games still to play. It's nice for now but there are big games to come still."