The Red Devils won 3-2 at Stamford Bridge, with Javier Hernandez grabbing the winner off the bench after Ivanovic and Fernando Torres had been sent off for the hosts.
Ferguson feels the right-back's dismissal was the turning point for United, who now sit just a point behind leaders the Blues in the Premier League table.
Asked if the encounter could have gone either way, the manager told Sky Sports: "Oh yes, absolutely, no doubt about that.
"We started the game so well, absolutely brilliant. Then just 10 minutes before half-time we lost our way a bit. We couldn't wait for half-time to be honest with you.
"I think the sending off of the right-back was the turning point for us. We would have won it at that point.
"We put Chicharito on and of course he gets the winning goal. I think the momentum was with us then."
Ferguson admits luck played a part after Hernandez's winner appeared to be from an offside position, adding: "They say the winning goal could have been offside, so that's a bit of luck I think we got."
The boss did not see a problem with Fernando Torres' controversial dismissal after he earned a second yellow card for what referee Mark Clattenburg adjudged to be a dive following a challenge by Jonny Evans.
"I thought he went down. I think Jonny may have just caught him a little bit," added Ferguson. "But you could carry on running, which he could have done, but he chose to go down.
"He could have carried on and scored, that's what I can't understand. And he's been already booked, so it's his own fault."