Striker Jamie Proctor headed in Crawley in front in the 14th minute at the Broadfield Stadium on Wednesday night, but Elliot Richards levelled after 83 minutes, before John-Joe O'Toole hit a dramatic winner for Rovers in the first minute of added time.
It was the third time the teams had met in the second-round replay after the abandonment of the previous match, following an initial stalemate.
Gregory said: "Bristol Rovers deserved to win, we did not.
"My boys have done fantastically since my arrival and have given their heart and soul in every game so it's important to keep an even keel.
"We need to stay calm after this result. It's time for a composed head, and for us to be composed for our match at Coventry at the weekend."
Gregory was disappointed his players did not work Rovers goalkeeper Steve Mildenhall enough in the second half.
He added: "It's a big learning curve. You tend to find out more when you suffer defeats and we have to take it all into consideration.
"Often when you're 1-0 up at half-time you go on to win but it was not the case in this game.
"But credit to Bristol Rovers, they played with men up front and stuck the ball in our half as often as they could. When we had the ball it came back too often.
"The players have done very well since I came to the club but they feel with this result they've let the fans down."