The Gills were two-up within 13 minutes as former AFC Wimbledon striker Danny Kedwell scored a quick-fire brace, both from the penalty spot.
Chelsea loanee Patrick Bamford scored a double to level things before Hollands had the last laugh just before the hour mark.
"I thought we deserved a point, but you don't always get what you deserve in football," said Robinson. "I don't think the players could have put anymore into it.
"The disappointing thing for me is that we started the game wonderfully well in the first five to 10 minutes. There was a blatant penalty on Greeny [Danny Green], he was booked for diving after the tangle with Joe Martin after five minutes and there was one on Patrick [Bamford] that was blatant.
"The problem you have with young players is that they want to impress so much, Tom's [Flanagan] eagerness to win the ball back cost him the first penalty. Then he makes a mistake with a pass for the second [penalty] and it was a rush of blood to the head. It's a learning curve for him, and he's got to learn very quickly.
"I couldn't have asked for anymore from the players. I can't be critical of the effort, the style of play, the appetite. The last 20 minutes, we outplayed them.
"We passed it, we created chances, and we looked fit. I want to take the positives out of it. It's a defeat, but the manner of the defeats so far, we come away from Peterborough and here asking: 'what's just happened?' We have just been on the wrong end of things that have prevented us from winning the football match."
The first penalty came after 11 minutes after defender Flanagan fouled Cody McDonald, and the same players were involved in the second penalty decision 120 seconds later - following which Flanagan received a straight red card.
Chelsea loanee Bamford headed in from Green's free-kick after 26 minutes, before he coolly converted from close range on the half hour mark to level things up.
Bamford nearly scored an impressive first-half hat-trick, but agonisingly for the striker he was millimetres away from diverting Izale McLeod's teasing cross past Gillingham goalkeeper Stuart Nelson.
Gills' winner came after 59 minutes as Charlton loanee Hollands latched onto Michael Harriman's looping cross to head the hosts to their first home victory since April 6.