New-signings Marcus Maddison and Luke James scored their first goals for the club but a double from Chris Dagnall early in each half and a clever flick by Bradley Pritchard earned Orient the win despite having Romain Vincelot sent off with 20 minutes to go.
"I thought we were the better team in the first half and played good football," said Ferguson.
"We lost a little bit of our discipline at 2-1 but with another five minutes we could have had an equaliser.
"Maddison was the most exciting player on the pitch. Every time he got it in a one-on-one situation there was a buzz around the place. He caused them problems.
"Luke James is lively. Those two were the main two we wanted and we have done good business.
"It's disappointing to lose. It was a real challenge to retain the trophy but it was the hardest tie we could have got.
"We've got to learn quickly - we were a little bit naive at times.
"But we look good and look like we have goals in us. The players will be disappointed and will feel we should have won."
Dagnall headed in Lloyd James' fourth-minute free-kick before Maddison drove in 12 minutes later from 20 yards after a short corner.
Dagnall smashed in at the near post just after the re-start before crossing to Pritchard to flick in on the hour.
Vincelot was sent off soon after for a second yellow card after tripping Maddison but Orient held on despite James tapping in right at the end.
Orient boss Russell Slade said: "I thought after half-time was our best spell. It was good build-up play for Dagnall's second goal.
"The final ball was there which we were asking for at half-time and a fantastic finish.
"We looked a threat from set-plays and it was a cheeky little goal from Pritchard.
"We showed a real resilience out there although I was disappointed to concede a second goal.
"The red card, there did not seem to be a lot in it, but when you're on a yellow you need to be more sensible.
"Not having a game at the weekend gives us time to get players up to speed.
"Any cup run can give you momentum and you can get confidence from it.
"It's always a tough game when we play Peterborough but I would swap this win for three league points and I know they will be in the shake-up. It was important to keep our run going."