Goalkeeper Tim Krul - brought on by Louis van Gaal for the spot-kicks finale - saved from Bryan Ruiz and Michael Umana to put the Dutch through.
They now face Argentina in the second semi-final in Sao Paulo on Wednesday.
Van Gaal made two changes to the Dutch side that began the 2-1 round-of-16 win over Mexico. With Nigel de Jong ruled out of the rest of the tournament having sustained a groin injury last Sunday, Wesley Sneijder took up a more central role as Daley Blind played wide on the left and Bruno Martins Indi returned to the rearguard. Paul Verhaegh dropped to the bench to allow Dirk Kuyt to switch to the right flank and Memphis Depay to come in as a left-sided forward.
Costa Rica were without centre-back Oscar Duarte after his red card against Greece last weekend, with Johnny Acosta replacing him in Jorge Luis Pinto's only change.
Holland dominated possession, but could not break through Costa Rica's well-drilled defence in the opening stages.
Los Ticos threatened in the 19th minute when Joel Campbell turned inside Martins Indi and sent in a low cross towards the box, which was cleared by Ron Vlaar with Christian Gamboa waiting at the far post.
The Oranje almost broke the deadlock in the 21st minute when Depay fed Robin van Persie in the left side of the box but his powerful attempt was kept out by the outstanding Keylor Navas, who also saved Sneijder's follow-up.
At the other end, Celso Borges could not get his head behind Christian Bolanos' dangerous free-kick.
Navas was called on again to keep out Depay's left-footed shot towards the near post and in the 38th minute, he palmed away Sneijder's curled free-kick towards the far post.
Seconds later, Navas rushed off his line to deny Van Persie following Arjen Robben's through-ball as the teams went to half-time with the score at 0-0.
Holland kept up the pressure after the break but Los Ticos defended deeply.
On the hour mark, Campbell demanded a penalty after being nudged in the area by Martins Indi but his appeal was waved away.
Five minutes later, Giancarlo Gonzalez's flying header from Bolanos' free-kick went over the bar.
Pinto made his first substitution one minute later, with Marco Urena replacing Campbell.
With 15 minutes remaining, Vlaar headed Sneijder's free-kick just over the bar.
Costa Rica then suffered a blow with Gamboa stretchered off and having to be replaced by David Myrie.
Holland came closer than ever when Sneijder's free-kick hit the post with Navas beaten.
Two minutes later, there was a scramble in the Costa Rica area and the ball fell to Van Persie but his close-range effort was repelled by Navas.
Van Persie then missed a golden chance to put his team ahead when he failed to get a touch on Sneijder's fantastic cross towards the far post.
The last chance of regulation fell again to Van Persie after Blind's cross eluded three team-mates. Yeltsin Tejeda, on the line, was able to divert his shot straight upwards and off the bar, with Stefan de Vrij eventually running the loose ball out of play.
Holland continued to press forward in extra time and in the 94th minute, Vlaar's header from Sneijder's corner was saved by Navas.
Van Gaal threw on Klaas-Jan Huntelaar for Martins Indi for the second half of extra-time - during which Costa Rica won their only corner of the entire match - but there was still no way past Navas.
There was still a chance though for Urena to win it but Jasper Cillessen denied him, and then Sneijder rattled the woodwork once again at the other end.
Van Gaal chose to replace Cillessen with Newcastle shot-stopper Krul for the shootout.
Borges and then van Persie both coolly slid home the first two spot-kicks, but Krul dived to his left to deny Ruiz which allowed Robben to give Holland the advantage.
Gonzalez and Sneijder hit the target, with Bolanos and Kuyt following suit.
But Umana - who scored the crucial penalty against Greece in the last round - was also denied by a Krul save, thus sending the Dutch through.