Fraser Aird had cancelled out Swankie's opener but Athletic rallied to dominate their League One rivals in the extra half-hour and book their place in the second round.
Rangers began sloppily and Gavin Malin had already tested Scott Gallacher when they fell behind in the ninth minute - Lee McCulloch's slip allowing Swankie to convert with aplomb.
Chris Templeton sent a header whizzing past the Rangers post as Forfar looked to double their advantage, but Ally McCoist's men improved slightly after the break and went close through Robbie Crawford.
The visitors continued to huff and puff in search of a reply but seemed destined to fall short as determined Forfar defending preserved their lead until the closing stages of normal time, when Aird pounced on a rebound to nod home.
That was as good as it got for Rangers, though, as they failed to make an impact in extra-time - Swankie capitalising on more indecisive defending at the death to meet Iain Campbell's cross with a match-winning header.