Roland Lamah was on target after just two minutes, with the Swans unable to add any further goals as they dominated the remainder of the first half.
Stephane Sessegnon levelled matters from just outside the box on 52 minutes, with Youssouf Mulumbu firing home the winner five minutes from time.
And midfielder Britton said: "We played really well in the first half and could have been two or three up but we were very poor in the second half.
"We invited pressure and when you do that the other team score.
"West Brom started strongly in the second half and we couldn't match it.
"And, as the match wore on, they got more belief and we got edgy.
"Scrappy play from ourselves helped them get their winner, even though he struck it well - but we should have got rid of it.
"In the first half we moved well, passed it well and created chances but we were a different side after the break.
"I suppose in the coming days we will watch the video and see where we went wrong.
"But we're in a relegation battle and must battle hard from now on. We've got nine cup finals."