The striker, who has joined Caley from Coleraine, says he turned down the opportunity to take on a Premier League challenge with the Swans.
A switch to South Wales would have presented the 25-year-old with a shot at Europa League football, but he feels his ongoing development will be better served in the SPL.
Allen told the Daily Record: "I had a couple of other options. Swansea showed an interest but once I spoke to Inverness I was really keen to go there and it seemed the perfect move.
"I'm relishing the move. It's a big step up from the Irish League but I will go there full of confidence.
"I never made a secret of the fact I wanted to get back in full-time football again. People questioned if I was good enough but hard work has paid off."
Allen came through the academy ranks at Bournemouth but was unable to force his way into the senior fold.