Goals from Samir Nasri and Sergio Aguero put City 2-0 up and although Demba Ba replied for the Blues, they held on to book a final date with Wigan.
City skipper Kompany said: "It's great. You don't get to play a lot at Wembley and when you win it is historic for us.
"Chelsea are a good team and you can see with the front players they have got you are always liable to concede chances.
"I think our strength is to be organised and to defend and we have been able to absorb it. They are very strong."
Victory over Wigan on 11 May will provide City with some consolation after their apparent failure to retain their Premier League crown.
"It is not a question of salvaging (the season)," said Kompany. "You never know how good your season is until the last game.
"We wanted to get trophies for our fans and bring back trophies to Manchester.
"If we can do it this season then we need to cherish it and we need to be happy about it because it is a big competition.
"You can always improve and we have still got a lot of the season to do that. We shouldn't look back too much now.
"I think anyone that participates in the FA Cup should take it seriously and that is what we have always done, not just today but in the previous rounds as well. It is a great competition."
Kompany expects the final to be one to savour with Wigan, although in relegation trouble in the Premier League, noted for their attractive and fluent style of play.
The 27-year-old Belgium skipper said: "I think this is a stadium, a pitch, that probably suits Wigan but it suits us as well.
"I expect it to be a good game because they know how to play ball and we do.
"For the fans it is always a bit harder when it is open but I do feel it will be a great game for the neutrals again."
City face Wigan in a final dress rehearsal at the Etihad Stadium on Wednesday but that match will itself still carry huge significance at both ends of the table.
City, trailing Manchester United by 15 points at the top of the table, are anxious to secure second place while Wigan are in the bottom three.