The Potters recruited highly-rated teenage goalkeeper Jack Butland and US international winger Brek Shea during the January transfer window.
But experienced duo Matthew Upson and Rory Delap left for Brighton and Barnsley respectively, leaving Kemp with worries prior to Saturday's visit to the Emirates.
"We've brought in two excellent young players but they are both for the future," he said.
"To be honest, the squad is unbalanced. We've been aware of that since the summer.
"We tried to correct it in the last window and didn't manage it, and the plan was to get through to this window and address it and unfortunately we haven't.
"So there is a bit of concern there. We let two senior players go out on the preface we were bringing players in so, in some respects, you could argue we're a bit weaker than it might appear from the outside."