A helicopter had to land on the pitch to airlift a man with a serious head injury to hospital.
A doctor said three other fans were hospitalised but none of their injuries were life threatening and one was treated for a minor injury at the stadium in the southern city of Joinville.
Groups of fans punched and kicked each other several times until police - hitherto notable by their absence - finally arrived and fired rubber bullets to contain the situation.
Police were not in the stands when the fighting began. A police official said about 80 private security guards were on hand to make sure the fan groups remained separated.
The match was important for both teams as Atletico Paranaense were trying to secure a spot in next year's Copa Libertadores and Vasco da Gama, four-time Brazilian champions, were battling to avoid relegation.
Fan violence has been rampant in Brazil this year, raising concerns ahead of the World Cup, and the match was being played in Joinville instead of Atletico's base in Curitiba because the club had been punished for fan violence earlier this year.
"This is deplorable," said Vasco da Gama coach Adilson Batista. "It's sad to see images like these just before the World Cup in our country. I'm shocked, this is not sport."
Television images showed players yelling at the fans and telling them to calm down. Some players put their hands on their heads as they watched the fight.
A few fans tried to leave the stands to find safety, including a young woman, while a ball girl was shown crying.
After the fighting was over, a police helicopter landed near one of the goals to airlift the most seriously injured fan to a hospital.
"We tried to tell the fans to stop because things would only get worse. We looked at the stands and there were no cops. There was nobody there to stop the fighting," said Paranaense defender Luiz Alberto.
"I've been playing for 20 years and I've never seen anything like this in person. We will have a World Cup in our country and we know these images will be shown everywhere."
The referee stopped the match just after 15 minutes, with Paranaense leading 1-0. Paranaense went on to win the game 5-1 thus securing a place in the Copa Libertadores and relegating Vasco da Gama.