Jimmy Smith headed Russell Slade's men ahead in the 35th minute of Saturday's fourth-round tie at the Liberty Stadium, but the Swans levelled on the stroke of half-time through Cedric van der Gun.
Yet the real drama was still to come as Tate diverted a cross from substitute Paul-Jose M'Poku into his own net to send the League One club into the last 16.
Swansea threatened inside the fifth minute when Scott Sinclair was given the time and space to strike a 12-yard shot goalwards, but Orient goalkeeper Jamie Jones made a fine save to turn the ball round a post.
Moments later the Swans broke forward again and Kemy Agustien hit a 25-yard shot which drew a fine block from Ben Chorley.
From the resultant corner, an inswinging delivery from Andrea Orlandi was greeted with a firm header by van der Gun from 12 yards which flew over the crossbar.
The hosts were largely dominant during the opening 45 minutes and Agustien hit a 25-yard shot which was blocked by Orient defender Terrell Forbes.
Sinclair then hit a shot over from distance but the visitors opened the scoring against the run of play 10 minutes before the break.
Joe Allen fouled Smith and Charlie Daniels swung in a free-kick from the right flank.
Smith arrived on cue to plant a firm header into the net from 12 yards and hand Orient a shock lead.
Yet Swansea responded in style and a spirited rally led them to drawing level on the stroke of half-time.
Allen crossed dangerously from the left flank and Van der Gun headed home from six yards out.
Orient boss Slade made a tactical switch at half-time, replacing Scott McGleish with Jonathan Tehoue, but Swansea came close to forging ahead shortly after the restart.
Van der Gun fired goalwards from 12 yards but Jones made a fine save to tip the ball over the bar.
Moments later, Darren Pratley took an inswinging corner and Sinclair was given the time and space to strike the ball goalwards from 20 yards.
His effort missed the target but the Swans continued to dominate and steadily upped the tempo as the half wore on.
Orlandi had a shot blocked by Smith before he succumbed to injury in the 63rd minute and was replaced by Angel Rangel.
Rangel looked to make his mark immediately and fired just wide from 20 yards before van der Gun missed the target from 12 yards.
Yet Orient rode the spell of pressure and began to pose a threat of their own in the closing stages.
And with two minutes remaining, M'Poku crossed dangerously and Tate inadvertently sent the ball past Yves Ma-Kalambay to send Orient into the fifth round.