The Greece international defender had been close to a transfer to Leverkusen, but after an approach from the Champions League finalists has had a change of heart.
Papastathopoulos, 24, joined Werder Bremen from AC Milan on a permanent deal last summer after spending the 2010/11 season on loan at the Weserstadion.
Voller feels that Dortmund have acted hypocritically after expressing their anger at Mario Gotze's summer move to Bayern Munich.
"We had an agreement with the player and also with Werder Bremen, too," Voller told Bild.
"Now Papastathopoulos has cancelled the deal and joined Borussia Dortmund. This is very annoying from our point of view.
"Everyone knows that they [Dortmund] were aggrieved by the news that Mario Gotze joined Bayern Munich, and then in this case they are hypocritical.
"I can smile at this. That's how things are in business."