Boro comfortably secured their passage to the second round of the Capital One Cup thanks to goals from Marvin Emnes and Emanuel Ledesma at Gigg Lane on Saturday.
"It's all right but I sat there at the end and thought it's still a bit young and inexperienced around the top end of the pitch," said Mowbray.
"We'll try and address that if we can.
"We do have experience elsewhere. We've got something to build on and some scope still to work over the next few weeks.
"It was a tough start to the season and you'd have to say Bury played well. They've got some good footballers and asked some questions of us.
"Concentration, apart from one occasion from a corner kick, was good, we looked solid enough and were always a threat going the other way.
"It was a bit uncomfortable in the last 15 minutes but in cup competitions you have to get through and we got by a difficult obstacle.
"Marvin (Emnes) has the ability to score goals. We've talked to him about not over-elaborating around the box and to shoot when he gets the chance.
"He did today and it flew in."
Middlesbrough never looked genuinely threatened after Emnes had opened the scoring with a superb 25-yarder.
Ledesma extended the lead after Trevor Carson spilled Nicky Bailey's shot before Mark Hughes reduced the arrears.
Bury saw Efe Sodje sent off late on to compound their woes but caretaker manager Peter Shirtliff still enjoyed his audition for the full-time job.
"I enjoyed it apart from the result," said Shirtliff.
"I'm quite comfortable in the role so there's no problems there.
"It's not difficult to plan ahead. I know the players here and I know the squad I've got.
"I'm happy with it at the moment and if I need to tinker with that I'll speak to the chairman and the board.
"At the moment, I'm quite happy with things.
"We were always in the game and we were disappointed to lose it. When you assess the game and the opposition we had, I think we performed creditably.
"Their first goal was a bit controversial, in terms of whether it was a free-kick or not, but we switched off a little bit.
"They took the quick free-kick and within seconds we were behind, so we need to learn from that.
"But there were plenty of positives and I think there were some great performances out there.
"In terms of the sending-off, I can see why the referee did it."
