Jones lost his job the day after the club lost 2-0 at Blackpool on Saturday, with the 57-year-old having guided his side to just one victory in 16 league matches this season.
Assistant manager Paul Wilkinson has also left Hillsborough.
Gray will take charge of the side for the league match against Championship leaders Leicester with Wednesday six points adrift of safety.
"Sheffield Wednesday can confirm that first-team coach Stuart Gray will take charge for Tuesday's Sky Bet Championship clash with Leicester City at Hillsborough," read a club statement.
"Following the departure of Dave Jones as first-team manager, the club can also confirm that assistant Paul Wilkinson has departed S6."
Former Nottingham Forest, Barnsley and Aston Villa midfielder Gray spent two years in charge of Northampton from 2007 and has also had spells as caretaker-boss of Aston Villa, Wolves, Burnley and Portsmouth.
Gray previously worked alongside Jones at Southampton and Wolves before becoming part of the coaching staff at Hillsborough last December.