Gazidis was speaking following a meeting of the general assembly of the European Clubs' Association (ECA), and he believes Europe's leading teams 'should not cave in to pressure from agents'.
Research by the ECA has shown that 14.6% of all money paid in transfers are currently being pocketed by agents.
FIFA's rules state 5% of transfer fees should be paid to the teams that trained the players between the ages of 12 and 23.
However, it is estimated that nearly £200m has not been paid out in accordance with this rule and that clubs in Africa and South America were the ones most likely to have missed out - unaware that this legislation exists.
Gazidis said: "Clearly there is a lot of money going to agents in these transfer transactions but I don't think we the clubs can abrogate responsibility.
"We are part of the problem as we are the ones agreeing to pay these amounts of money.
"There are unusual pressures in the system which lead to agents having significant leverage.
"Players should be making more money and that money not going to agents.
"If I were a player or players' union, that would be a really substantial concern to me because it means my clients as a union and me as a player are losing money.
"More invidious are conflicts of interest that often exist in these transactions with agents, and that's something that clubs and players should be concerned about."