The 24-year-old, who spent last season on loan at the John Smith's Stadium and scored 14 goals in 37 appearances, has signed a three-year contract with the Championship side.
Huddersfield manager Mark Robins said: "James' arrival is a significant signing for the club and one that I know our supporters will be very pleased with after his excellent debut season in a Huddersfield Town shirt.
"Equally, I'm delighted that we have secured his services. The exciting thing is I believe there is even more to come from James.
"He is only 24 and although he already has excellent qualities in terms of strength, pace, work rate and much more, I believe he will have learned from the games he played last season and we will benefit from his knowledge and understanding of the style of football we want to play here."
Birmingham-born Vaughan began his career as a trainee at Everton and went on score nine goals in 49 appearances for their first team.
He had three loan spells in the Championship with Derby County, Leicester City and Crystal Palace, before ending his loan association with Everton in 2011 when he moved to Norwich City.
He joined the Terriers in late August 2012 on a season-long loan and became a firm favourite, winning the club's player of the year award.