Forster had broken former Aberdeen goalkeeper Bobby Clark's 43-year-old Scottish league record on Saturday and it took something special to finally beat him after 1,256 minutes as Hayes struck a spectacular long-range effort.
Having conceded for the first time in 14 Scottish Premiership games, the England international shipped another goal four minutes later when Adam Rooney sent a diving header past him from Niall McGinn's cross seconds before the interval.
Substitute James Forrest's excellent finish in the 61st minute got Celtic back in the game following a spell of pressure but they had been up against it from the 11th minute, when Virgil van Dijk was sent off for bringing down Peter Pawlett.
And the Dons held out to inflict Celtic's first league defeat for 10 months since they lost at Motherwell.
Steven MacLean marked his return from a four-month injury lay-off with two goals against hapless Motherwell in a 3-0 win for St Johnstone at McDiarmid Park.
MacLean, who was returning from knee surgery, grabbed his first of the evening when goalkeeper Gunnar Nielsen parried a ball into his path after saving a Stevie May free-kick.
He then scored his and the Saints' second when Brian Easton's cross from deep on the left found him nipping in ahead of Shaun Hutchinson at the back post in a first half dominated by the home side.
The points were secured for the Perth side late on when Gary Miller ran down the left unchallenged before cutting inside and lashing an unstoppable shot into the net.
Goals from David Raven, Billy McKay and Marley Watkins handed Inverness a comfortable 3-0 victory over Highland rivals Ross County on Tuesday night.
County looked a shadow of the side that beat their local rivals on New Year's Day as Inverness ran out deserved winners.