Midway through the opening 45 minutes, Kevin Nolan went down under the challenge of Barry, who was the last man just outside the area, but Moss waved play on despite the Hammers' protestations.
Moss kept his cards pocketed and the hosts eventually went on to claim the three points thanks to second-half substitute Romelu Lukaku's late strike.
Allardyce was mystified by the decision and says that he had the same view as the referee at the time.
"I saw it at the time and I saw it from behind where John was positioned so I was in exactly the same position as John Moss, 40 yards behind him so I could see it," he told Sky Sports.
"Apparently he said to Kevin (Nolan) that he couldn't, so it's beyond me to say he can't see it when I can and he can't.
"It's clear that Gareth Barry has got both arms around Kevin Nolan, that's why I was jumping up and down like a lunatic at the time, moaning at the fourth official saying 'why the hell has he not given us a free-kick?'.
"Then having a look at it, it's Kevin Nolan through on goal and Leighton Baines was well behind Gareth Barry.
"Gareth Barry knows he's got to pull Kevin down because he's got behind him. He then knows that Leighton Baines is not going to catch him up and that's why he pulls him down. But not only was it not a free-kick, he didn't give anything.
"I suppose he's going to say and hide behind the fact that he's said he couldn't see it but it's a bit much when I can see it when I'm in the same position as he is - no obstructed view, clear view of the incident.
"Like I said, I was directly behind him and I could see it from the distance I was at and those are big turning points in game but they're out of our hands. We can't affect what the referee does on the field, we can only affect our own performance."
Despite the controversy, Allardyce was proud of the showing from his side and admits that a draw would have suited him.
"It looked like it was probably going to pan out to a 0-0 draw which would've obviously been very good for us, picking up another point and another clean sheet," he said.
"But unfortunately, in the end, that wasn't to be so we're disappointed but we've got to bounce back from this.
"We make life extremely difficult for opposition players and then of course when we were in possession, I thought we got a lot of good possession, particularly in the second half in the final third.
"Today we didn't do enough with it as we've shown in the last five or six games. That's why we've scored goals and won by two-goal margins.
"Today we didn't find the final third play we've been finding and hence we've ended up losing that 1-0."
Allardyce also cleared up his thinking behind swapping Carlton Cole for Andy Carroll during the first half, stating tactical reasons for making the change.
He said: "I was going to bring him on at half-time but I just felt that Carlton wasn't holding the ball up enough for us and that's why we couldn't get out of our half in those first minutes of the first half.
"I needed better hold up play and I think that playing into Andy, he gave us that and that's why we got going forward that little bit more."