Billy McKay opened the scoring with a penalty for Caley Thistle after nine minutes. Iain Vigurs equalised before the break but Inverness booked a fifth-round trip to Kilmarnock courtesy of McKay's 54th minute winner.
Adams said: "Inverness get a penalty in the first half, but then Grant Munro doesn't get awarded one in the second half. The decisions haven't gone for us.
"We were the better team, We had more possession and opportunities. Mark Brown had one save to make, but has picked the ball out of the back of the net twice.
"We've had a couple cleared off the line and their goalkeeper has made saves.
"It's hard to take because we were camped in their half for the whole second period."