Andy Robinson gave the visitors the lead just before the break, converting from the penalty spot after Harlee Dean had fouled David Amoo in the area.
Brentford hit back when Dean redeemed himself, getting ahead of his marker to glance home Jake Bidwell's near post free-kick.
But Adam McGurk ended Brentford's 15-game unbeaten run when he skipped past two defenders to fire in a low angled drive for the winner in the 66th minute.
Rosler said: "We made mistakes for both goals and they punished us, but we had 23 efforts on goal to their six and on another day one of the shots that was blocked goes in and you win.
"I'm not panicking, my coaches are not panicking and the players aren't panicking at the run ending because we are not Barcelona.
"We were beaten by a very good side. They were very compact and worked hard.
"They did their homework, but the second-half performance was very good and we will take that into our next game against Leyton Orient when we will have to work our socks off to get a result. I thought we were worth a draw."