Banton, named npower League Two player of the month for March, recently had his loan spell at Home Park extended until the end of the season but Palace boss Ian Holloway has decided to bring him back to the Championship promotion chasers.
Argyle manager John Sheridan admits losing the 20-year-old is a "blow" for his side, who are three points above the drop zone with three matches remaining.
"It's a big loss because he gives us a spark," Sheridan told the club's official website.
"He does things other players can't do. He's off-the-cuff and always a threat. It's a blow and I think Jason would like to have stayed, but it's not his decision.
"Credit to him - he's been getting a bit of notice; Palace aren't winning games, so they've recalled him. He's their player and I respect that. They've helped us out and he's done really well for us."
Banton scored six goals in 14 appearances for Plymouth, with three of his goals coming in March.