Leverkusen qualified for the group stage of the Champions League by finishing behind Bayern Munich and Borussia Dortmund in the league, but they have lost Andre Schurrle and Dani Carvajal this summer.
Heung-Min Son and Giulio Donati have been brought in as replacements, but Rolfes is worried about a possible drop in quality when the club actually need more strength in depth next season.
"The quality of the team has got to get better than last year if we want to be successful in the Bundesliga and the Champions League, or else it will be tough," he said in the Rheinische Post newspaper.
"We have lost players who we could rely on to be able to play consistently at a high level, but you never know what to expect with new players."
Leverkusen made Emir Spahic their third new signing this summer on Friday with the Bosnian central defender increasing competition for places at the back.
"He is going to help us realise our goals in the Bundesliga, DFB-Pokal and, of course, also in the Champions League," said the club's director of sport Rudi Voller.