A mistake by goalkeeper Jack Butland allowed Moshtagh Yaghoubi to give Finland a 15th minute lead from a long-range free-kick.
But the visitors recovered well, equalised in the 67th minute through Saido Berahino, and continued to impress Southgate even after skipper Andre Wisdom was red carded with 16 minutes left.
Southgate said: "We said at the beginning of the week that we won't go into every game and not make mistakes or not go behind, so it's how we respond to those things that matters.
"We've got to keep doing the right things and we kept passing and probing. We kept doing that when we went down to 10 men as well, which was really pleasing.
"I thought (midfielders) James Ward-Prowse and Tom Carroll were terrific. Even when we were down to 10 men, they kept probing and opening them up, and also did a disciplined job defensively when we needed that cover."
Southgate was also happy with the displays of forwards Berahino and Harry Kane.
West Brom striker Berahino has scored in both of his U21 appearances so far and Tottenham's Kane did well after coming on in the second half.
Regarding the change, Southgate said: "You think about doing it at half-time, but then you think to give it five or 10 minutes.
"But we felt we needed to give them something else to think about defensively.
"Saido had been a goal threat and we know that Harry's a goalscorer, so maybe that change of shape gives the opposition something to think about.
"It paid off for us, as we had that spell when Harry nearly gets the header and we hit the post when we were down to 10 men."
England have four points and sit third in Group 1, having played half as many games as leaders Finland, who are on eight points.