Adam Johnson's sumptuous 33rd-minute free-kick handed the Black Cats the lead, but it lasted just nine minutes as boyhood Sunderland fan Matty Robson lashed home a shot off the underside of the crossbar with Wes Brown unable to clear on the line.
However, the home side needed just five minutes of the second half to regain their advantage when Carlisle skipper Sean O'Hanlon turned Johnson's firm cross past goalkeeper Greg Fleming, and substitute El-Hadji Ba's last-minute strike wrapped up the win.
His afternoon on Wearside reinforced Kavanagh's belief in the magic of the FA Cup and he said: "It's huge. It's so exciting for a club like ourselves to come to a stadium as big as this with a fanbase as big as it is.
"For us then to get behind it because of the location and go and support the game and the team as they have done, and then always to feel that you have that chance of potentially going through and looking to see who you are going to draw.
"When we beat Brentford and the draw was made for Sunderland, the excitement around the city was huge.
"Paul Lambert has spoken about probably diminishing the FA Cup a little bit.
"I can totally understand where he is coming from as a manager because they need to stay in the Premier League for the financial gains, but the romance of the cup has always been, and will always be fantastic, and I hope we don't lose that."