The Black Cats enjoyed good spells of play, taking a 1-0 lead into half-time thanks to Fabio Borini's opener, but City came back to overpower them, with goals from Yaya Toure, Samir Nasri and Jesus Navas clinching the trophy.
O'Shea had previously lifted the trophy on three occasions during his time at Manchester United and he was regretful at not being able to add to that tally against his former club's city rivals.
"We're terribly disappointed, the only way you enjoy a cup final is by winning it," he told Sky Sports.
"We started the game exactly how we wanted to. We counter-attacked great, scored a fantastic goal, had a couple of great chances."
Toure's equaliser was the pick of the strikes as he swept a first-time effort into the top corner from long range, but O'Shea felt that there was a foul in the build-up to the goal.
He said: "They've scored a great goal to get back into it. We were hoping to get a free-kick on the edge of the box but it wasn't to be, we should've maybe got out to him.
"He stuck it in the top corner from 30 yards out and then we probably lost a bit of our organisation for a few minutes and it cost us for the second goal.
"We might've had a few half chances but City obviously finished the game very strong.
"As I say, the only way you can enjoy a cup final is by winning it. It's not a nice place now and we have a huge test for the rest of the season. We've got to pick ourselves up now and I'm sure we will.
"We had some great counter-attacking play and could've been a bit more clinical but look, City are a good team, we know that. Credit to them, they deserved the win."