The defender's failure to control a ball on the edge of his own penalty area gave Michu the chance to open the scoring in the first half against the run of play.
Chelsea, who started with Fernando Torres instead of January signing Demba Ba, dominated but Danny Graham made it 2-0 in injury time when pouncing on Ivanovic's blind back-pass.
The European champions had a claim for a penalty, which saw Ba booked for diving, immediately before Graham's goal in a dramatic conclusion to the game.
But interim boss Benitez heard his team booed off at the final whistle and they must now overcome two away goals in the second leg.
Chelsea's fans spent a large chunk of the match berating Benitez for his failure to bring on Ba, who scored twice on debut last weekend, and they also took umbrage with the interim boss' failure to bring on Frank Lampard until the 70th minute.
The Blues dominated the game, but lacked a killer touch up front and they paid the price in the 39th minute when Ivanovic's blunder gifted Michu the chance to score his 16th goal of the season.
David Luiz, by far Chelsea's best player, went close with two efforts just after the break but the home fans continued to vent their anger at Benitez for not introducing Ba or Lampard.
One fan had to be restrained by stewards as he headed in the direction of the Chelsea bench to remonstrate with the former Liverpool boss.
Another fan invaded the pitch to provide the Blues fans with some entertainment. The rest of the home crowd continued to demand Lampard's introduction but the calls fell on deaf ears.
Lampard injected life in to the Chelsea team but Torres continued to perform poorly. The striker found space on the edge of the box, but could only head weakly in to Gerhard Tremmel's arms.
Ba was sent on in place of Torres in the 81st minute and hurtled through in injury time, going down after Tremmel failed to collect the ball in the penalty area, but the referee did not give a penalty and instead booked the former Newcastle United man for diving.
Chelsea's night then turned from bad to worse with one minute of added time left. Ivanovic's second terrible back-pass allowed Graham to pinch the ball off Turnbull before slotting in to an empty net while Ba's later disallowed effort for offside only added to the frustration.
|Chelsea||Team Statistics||Swansea City|
|0||1st Half Goals||1|
|7||Shots on Target||3|
|13||Shots off Target||1|
|59.4||1st Half Poss.||40.6|
|71.8||2nd Half Poss.||28.2|