Scott Wootton scored on his Posh debut following his arrival on loan from Manchester United, but headers from Greg Halford and Elliott Ward gave Forest a 2-1 win.
Ferguson said: "To lose two goals and the game to two set-pieces is very disappointing because I thought a draw would have been a fair result.
"I am repeating myself a lot this season but I told the players afterwards that it was another game we should not have lost but it was frustrating to concede the way we did.
"Forest don't have too much height in the team for a Championship side but their two tallest players have got the goals from headers."
As for the officials, he said: "Things went against us at times. I have seen the first goal and I'm still not convinced that it was over the line but the referee's assistant decided from 40 yards away that it was.
"We should have also had what I thought was a blatant penalty but there's no point in talking to officials about things like that."