Boro had midfielder Anthony Grant sent off, as Sanchez Watt bagged the U's winner on 55 minutes to end Stevenage's unbeaten run.
Smith said: "The referee had a bad afternoon and he compounded it with the sending off.
"I'm frustrated and that's borne out of the fact that it was a very competitive game and, with 11 men on the field, we'd have walked away with another point on the board, at least.
"With 10 men and a quick goal, it was a mountain to climb. He's made life difficult for himself by booking just about every individual that's made a challenge in the first half.
"Professionally, our guys have got to be more mindful and more appreciative of being on the field with such a pedantic individual. I'm not sure Anthony was so he's put himself in the firing line again and allowed the referee to do exactly what he wants to do, which is get the spotlight on him and show that he's dictating."
