It had finished 2-2 on the night and 4-4 on aggregate after extra-time, but Tom Huddlestone and Emmanuel Adebayor missed in the crucial shoot-out to send the north Londoners out at the quarter-final stage.
Clint Dempsey's opener was cancelled out by Mohamed Salah before Aleksandar Dragovic gave the Swiss side the lead early in the second half.
However, Dempsey came to the rescue again as he restored parity with seven minutes remaining. But, defender Jan Vertonghen was shown a straight red card in the final minute of normal time for bringing down Marco Streller.
The extra 30 minutes failed to produce a goal despite Mohamed Elneny coming close when he struck the woodwork.
Dempsey handed Tottenham the perfect start after 23 minutes when he took advantage of Aleksandar Dragovic's poor clearance to round Yann Sommer before coolly slotting the ball into the empty net.
However, the lead lasted just four minutes as Moussa Dembele gave the ball away to Marco Streller and he squared the ball for Salah to toe-poke his right-foot drive past Brad Friedel.
Murat Yakin's side were back in the lead soon after the break as Fabian Schar's flick was parried out by Friedel to Dragovic, who couldn't miss.
But back came the visitors once again as substitute Huddlestone delivered a well-crafted chip that Dempsey drilled under Sommer.
Vertonghen made life tougher for his team when he tripped Streller as he ran through in the 90th minute and received a straight red card.
Spurs held on for penalties - but Huddlestone missed Tottenham's first spot-kick to give Basel the advantage and Adebayor then skied Spurs' third while Schar, Streller and Fabian Frei all scored before Marcelo Diaz bagged the crucial winner.
The defeat means Spurs have now lost their last seven successive penalty shoot-outs while Basel reached the semi-finals of a European competition for the first time in their history.