The Robins dominated the early chances at Whaddon Road with Steve Elliott and Kaid Mohamed going close before Shaun Harrad flicked home a low Jermaine McGlashan cross for the 16th-minute opener.
Jake Taylor glanced a header wide for Cheltenham but Hereford levelled with their first attack of the game on 20 minutes. Josh O'Keefe blasted home from 10 yards after his initial header had been palmed away by Robins keeper Scott Brown.
Mohamed was twice denied by Bulls keeper James Bittner after the break while Will Evans went close to snatching victory for the visitors.
"We didn't deserve to lose but didn't do enough to win the game," Robins boss Yates admitted.
"I thought we started really well. We moved it quickly and we moved it with a purpose but once they scored we didn't do it enough. They sat deep and made it difficult for us.
"In the second half, we didn't really play with enough quality, enough tempo or pace to our game to create enough chances. We weren't as good as we wanted to be but we were the team doing all the pressing.
"We were trying to make things happen. They worked really hard and all credit to them for that.
"No-one in our dressing room thought it was going to be easy but it's disappointing because we didn't apply ourselves. They played well but we can play better and we need to do that in the replay.
"We are a disappointed dressing room and they are doing a lap of honour but we are still in it."