The Sweden left-back, who can also play in midfield, returns to the Premier League on a four-year contract after Rovers accepted an undisclosed offer.
Olsson joins Javier Garrido, Ricky van Wolfswinkel and Nathan Redmond in signing on for Chris Hughton's side.
And Hughton told the club website: "We're delighted with the addition of Martin. He's been playing in England for seven years and is well acclimatised to the Premier League.
"He can also play anywhere on the left hand side so he brings us good flexibility in that sense, plus at 25 he's at an age where he has some experience and also some time to improve.
"At this time it's about improving the squad that we have and Martin certainly does that."
Olsson, who arrived at Ewood Park as a 17-year-old eight years ago, leaves with a record of six goals in 143 appearances.
Norwich have also signed Stoke keeper Carlo Nash on a one-year deal to add to their depth in goal.
With Jed Steer gone to Aston Villa and Declan Rudd heading to Preston on loan, Nash becomes the third-choice at Carrow Road and provides backup to John Ruddy and Mark Bunn.
Hughton added: "It is important to us to bring in a 'keeper of great experience and a player that is used to living life in the Premier League.
"The move also allows us to send Declan Rudd out on an extremely valuable season-long loan to Preston where he can get some first-team football under his belt."