In the dying moments of the game, with Reading desperately seeking an equaliser, Adam Le Fondre tumbled in the box under a challenge from Ryan Shotton but Oliver waved away the protests.
Earlier, Robert Huth and Cameron Jerome had given Stoke a 2-0 lead, which Adrian Mariappa halved for the away side in the 83rd minute.
In recent weeks, Reading have come back to claim points with late goals and McDermott feels his side were wrongly denied another chance to do so on Saturday.
"Once we got to 2-1, I thought we would get it (a draw), and unfortunately the ref has missed a penalty at the end, the last kick of the game," said McDermott, whose side remain 17th in the table, two points clear of the relegation zone.
"He got hold of the ball, flicked it by him and he caught him. It was a stonewall penalty. I've just gone in to see him (Oliver) and he didn't see it that way, so that's fine, but I thought it was.
"Sometimes as a manager you get those things - with the last kick of the game, it would have been great to get the point.
"We hung in there and showed great resilience and desire. This is not an easy place to come to, which everyone knows."