The Scot was shown a straight red card in the 5-1 home defeat to Leyton Orient on the opening day and has watched the Cumbrians concede another 11 goals in their next three matches as he sat out a three-game ban.
"It's been very hard to watch from the sidelines these last few weeks," Miller told the club's official website. "There's obviously nothing much you can do.
"You can give as much positive encouragement and input as possible before a game and during the week, but there's nothing you can actually do out on the pitch.
"I've let the boys down and it feels like I've let myself down."
Manager Greg Abbott is fearing for his job following last week's 4-0 home defeat to Coventry and Miller, expected to return to the starting line-up at Colchester on Friday, added: "There were a lot of down bodies - beaten, defeated - in the dressing room after the Coventry game, but we have got to pick ourselves up.
"I just want to get back to it now. I've only had 30 minutes and I really am distraught about that."