The clash at the KC Stadium, which the hosts won 2-0, ended on a sour note as Alex Bruce was dismissed after the final whistle for lashing out at Lee, who appeared to have caught him with an elbow to the face moments before.
After the match, Steve Bruce, Alex's father, revealed they would appeal the dismissal and then tore into Lee, claiming his actions were 'cowardly' and insinuating that he had deliberately got his son sent off.
However, Lee defended himself by claiming Alex Bruce's red card was fully vindicated.
He told the Huddersfield Examiner: "I went to close the ball down off another player and he came running into me to stop me doing that.
"I put my arm up to protect myself. He reckoned I elbowed him, but if I caught him - it was his fault.
"Why was he trying to run across me? I'm not trying to close him down. Then he reacted afterwards by flying into me."
Despite protesting his innocence, Lee fears his reputation could lead to a ban.
He added: "I will probably get punished by the FA - it's me, isn't it? It will definitely be my fault."