Bees boss Uwe Rosler has moved for the 20-year-old striker after summer signing Farid El Alagui was ruled out for the long term with a knee injury.
Spencer has played just twice for the Terriers but enjoyed a successful loan spell with Cheltenham last season, scoring 10 goals in 27 League Two appearances.
Sporting director Mark Warburton told the club's official website: "Jimmy is a talented young player.
"He has performed well and scored goals in League Two and is producing impressive performances at development squad level.
"We had to fight off stiff competition from other League One clubs to get him to Griffin Park. Everyone is looking forward to working with Jimmy and seeing him in a Brentford shirt."
