Brown's men looked sluggish following their winter break, and failed to take advantage of their chances.
Former Caley Thistle forward Jonny Hayes missed two opportunities for the Dons, and Brown feels those proved to be costly.
Brown said: "It was a poor performance from us, but all the credit should go to Inverness, who took their chances very well.
"Jonny Hayes had great chances for us in each half, but couldn't take them, and that was the difference.
"If you don't take your chances and you defend in the slipshod manner we did then that's what happens."