Grayson led Town back to the Championship at the end of 2011-12 but results have not been positive this season and they sit in 18th place, seven points above the relegation zone.
The 4-0 defeat by Watford last weekend appears to have been the breaking point for Grayson, with Town's only win since 10 November having come in the third round of the FA Cup against Charlton.
Chairman Dean Hoyle told the club's official website: "In making this decision, we have thought long and hard. It has been made with both the short term and long term needs of the Club in mind.
"We brought Simon in with the immediate aim to lead us to promotion and he did that at the first attempt. It is an achievement for which we are grateful.
"We knew the transition to Championship football would be difficult and our targets for this season have been realistic, with retention of our status the priority.
"However, having worked so hard to get back into the second tier we have to do everything we can to consolidate that position. After exceeding all our expectations in the early part of the season, our results and performances as a team and from individual players have tailed off alarmingly.
"Worsening displays have accompanied a run of just six points and no wins from the last 12 League games, including some very heavy defeats, culminating at Watford last Saturday.
"Whilst we would all have hoped for even some modest signs of improvement to begin to emerge as the weeks have passed that has not been the case. This has made the position untenable.
"We will now seek to recruit a new manager who can, in the short term, lead and reinvigorate the team and players for the remainder of this season and also drive us forward towards our long term aims."