Steve Jennings' fine strike in the 84th minute handed Rovers all three points after the Magpies had first broken the deadlock and then battled back to make it 2-2 just after the hour.
Ronan Murray fired Notts County in front in the 14th minute but two goals in the space of three minutes from Jason Koumas and Ryan Lowe put paid to their fine start.
Alan Sheehan then missed a penalty on the half hour - although he made amends after the break with a fine free-kick to make it 2-2 for League One's basement boys before the late twist but Derry was encouraged.
"It's fine lines when you get to this stage of the season. If I'm honest I was highly impressed with the performance that we put in," said Derry.
"I felt that we spent a large amount of time putting pressure on Tranmere but when it gets to the brutal critical points of the game we let ourselves down and we paid the price.
"When the third goal went in I felt we were on top, I thought we had a good amount of pressure, we sustained pressure and we looked sharp in behind the back four.
"But one error where we couldn't clear our lines properly and a great strike and it's game over. It's a damaging defeat but I tell you what it isn't - it isn't the end of the world.
"There are nine huge games to go, still 27 points to play for. I'm confident we can stay up; all the while we're not requiring snookers I'll be forever confident."