Aden Flint and Jay Emmanuel-Thomas strikes were cancelled out by Nicky Ajose efforts before Nile Ranger clinched a dramatic winner with three minutes remaining.
The defeat leaves Sean O'Driscoll's men in the bottom four in League One, despite being one of the favourites for promotion.
O'Driscoll said: "You have got to give the home side some credit, they believed they would score and we were three on three a lot of times.
"I watched a lot of League One last season and trust me, it's a far better division this season then it ever was last year.
"We are trying to build on something that has never been built on, this season was always going to be a transitional season for us, the fans and everybody at the club.
"For an away performance against a side that are confident, it was pleasing."