Graham joined Sunderland from Swansea City during the 2013 January transfer window, but he has yet to score in 13 appearances for the club.
He netted just once in 20 games on loan at Hull City during the first half of last season before rediscovering some form at former club Middlesbrough in the second half of the campaign.
Middlesbrough have held talks over a permanent deal for the 28-year-old, who has been left out of Sunderland's squad for the pre-season trip to Portugal.
With Sunderland hoping to bring Fabio Borini back to the Stadium of Light from Liverpool, Poyet suggested the opportunities for Graham will be limited.
"Danny has had a spasm in his back," said Poyet in the Sunderland Echo. "He hasn't trained for five days.
"I was trying to play him in one of the three games this week. But I'm not taking the whole squad to Portugal. There are people who are going to stay here. Danny is one of them.
"As everyone knows, we've got Fletch, Jozy and Connor, plus Fabio (Borini) if he comes in. It will be very difficult for Danny to be part of that group."