Maloney picked up the injury on international duty with Scotland and has not played for Wigan since the 2-0 defeat to Leicester City in September.
He is certain to miss his country's November friendly internationals with the United States and Norway but Coyle expects to be without him for several months thereafter.
Coyle said: "When my team and I arrived this summer, Shaun had already suffered from an existing hip injury, but had chosen to play through those problems as we all thought they would be manageable over the course of the season.
"However, when he was on international duty with Scotland he picked up a back injury which stemmed from the previous hip problem and when he returned it was clear he wasn't himself on the pitch.
"If I knew it was going to be three, four or five months then I could tell you but we won't know that until the surgeon opens him up and sees the extent of the damage.
"As it is, I think it's certainly fair to say it will be a minimum of three months, and obviously when the surgeon opens him up, there is a chance it could be longer than that.
"It's a blow to lose such a top player, but we still have tremendous quality in our ranks and players who are motivated towards doing well for Wigan Athletic."