Gilmour and Ross County chairman Roy MacGregor voted against the changes, which would have seen the league structure cut to three divisions of 12, 12 and 18 teams.
Milne said before the vote that the changes were needed to ensure the game survived and he gave an emotional interview to journalists after leaving Hampden Park in the wake of the 10-2 vote.
He said: "This is going to have major implications for Scottish football.
"These are going to start flowing through in the short-term and I can see other implications in the medium-term and it is going to potentially seriously damage the game in the long-term.
"What was on the table for everyone to consider today was back the plan, an opportunity to move Scottish football forward. If it doesn't deliver to the extent that 10 clubs genuinely believe it will deliver, then the opportunity is there to re-look at it down the line within two to three years.
"I'm very angry and I think today we have let down Scottish football. St Mirren Football Club have got to go and pose the question 'why are they prepared to put the rest of Scottish football in jeopardy just because of their own selfish interests which they have been unable to explain in reasonable terms to people around the table today.
"There were tense periods within the meeting and a lot of honest debate went on, which is needed. The decision has been made, we've got the current structure and we've got to look what we can do to improve that to find a more effective way forward.
"This is a massive, massive opportunity that has been blown, particularly on the whims of one individual club. I think everyone else was angry at the attitude that he (Gilmour) adopted. He's not prepared to look at the rest of Scottish football.
"We all have our responsibility when we are in that room for our own clubs, but we have a wider responsibility to the SPL as an organisation to have an interest in and wider Scottish football - none of that was displayed by Stuart Gilmour and St Mirren today."