The Gunners were sunk by two late goals at home to Swansea on Saturday, which was greeted by boos from the home fans at the final whistle.
Arsene Wenger's side are currently sitting in 10th position in the table - five points adrift of the top four spots - but 15 behind leaders Manchester United.
And with disappointing draws against Aston Villa and Fulham in recent matches, Spanish international Cazorla believes Arsenal must start taking maximum points.
"We don't know exactly what the problem is," Cazorla told The Mail on Sunday.
"It's true that we're inconsistent. I believe that to challenge for the Premier League, those games can't get away from us.
"Take the Fulham game, we were 2-0 up. You can't then be drawing within five minutes. Those points aren't coming back.
"And the other day against Aston Villa, there's a game which I believe we really had the desire to win, because these are points that we can't let get away.
"Then you see City, United winning on those days. That's the difference. To fight for the title, it's about taking points from those teams.
"It's a little bit frustrating. We're a good team, good players, training well. But there are moments in those games when we don't know how to control them.
"There's a significant gap between us and United and City. They've got away a bit. We have very little margin for error."