Bury managed to claim just their second clean sheet of the season, while Newport are ninth after losing once in their last eight league games.
"We knew where Bury were in the league, but they've only lost once here this season so it's a tough place to come," said Edinburgh.
"Any point away from home is a good one so we're delighted. The shape second half and the clean sheet were very pleasing.
"We didn't acquit ourselves well enough first half to combat them but sometimes you have to give credit to the opposition and Bury certainly came at us.
"We brought Chris Zebroski on at half-time because we needed to get an extra body up the field, we were being out-numbered there, and it certainly gave us some impetus second half.
"As the game wore on I thought we may nick the three points, which might not have been deserved, but our start and the way we finished was pleasing.
"After 10 games we're right in among it so I can't praise the players and the supporters enough. It's a really good start for us."
Marco Navas, brother of Manchester City winger Jesus, was handed his first Bury start and sliced an early shot wide.
Anton Forrester's cross then found him unmarked, but the Spanish winger somehow headed wide.
Newport threatened with Danny Crow and Lee Minshull going close, while Adam Chapman shot straight at Bury keeper Brian Jensen after the break.
Forrester headed over while Soares had one shot saved by Lenny Pidgeley and sent two off target.
Yet Newport went closest to a late winner through Minshull and Chapman.