Holland beat defending world champions Spain 5-1 in their opening match, but the Oranje had to come from 2-1 down to win 3-2 against the battling Socceroos to put one foot in the knockout stages ahead of their concluding group game with Chile.
"I'm not relieved, because I believe in the second half we played better than Australia," said Van Gaal.
"In the first half they played better, but in the second half we had five, six, seven chances.
"I expected us to win. When (Australia's Mathew Leckie) missed that chance, I said to Patrick Kluivert, 'We're going to win this match."'
Depay's swerving 30-yard strike, which deceived Australia goalkeeper Maty Ryan, came seconds after Leckie's missed opportunity and settled an end-to-end contest which came alive when Arjen Robben ran at the Socceroos defence and put Holland in front.
It was a sublime strike which was almost immediately surpassed in quality by a goal reminiscent of memorable strikes from Dutchmen Marco van Basten and Robin van Persie.
Right-back Ryan McGowan drilled a cross-field pass over the Dutch defence and Tim Cahill's exquisite left-footed volley went in off the bar in what is sure to be one of the goals of the tournament.
Van Persie - booked to earn a suspension which rules him out of the contest with Chile - swiftly cancelled out Mile Jedinak's penalty before Depay's winner eliminated Australia.
Man of the match Robben said: "We've seen so many great goals at this World Cup. It's great for everybody, great for the fans, and we saw (some more) today."
Robben praised Australia's performance as they pushed Holland all the way.
"They played really aggressively," he said. "They played like it was their last chance.
"We were a little bit under pressure, but we gave the ball away too easily.
"But we made big step today. We have maximum points. Goal difference can make a difference, so the first game was like an extra point for us."