Chris Gunter's own goal and Matty Fryatt's late penalty gave the Tigers all three Championship points at the KC Stadium.
Hull, whose top-six ambitions were all but dashed by a miserable sequence of five successive defeats last month, are unable to catch sixth-placed Cardiff, who gained a point from a 1-1 home draw with Leeds earlier in the day.
Nick Barmby's Tigers have won three and drawn one of their last four matches, but a second straight win at the KC Stadium in their final home game of the season was too little too late.
Forest, whose safety was assured last week, went behind against the run of play when Hull defender Liam Rosenior's shot came back off the post and rebounded into the net off Gunter in the 58th minute.
Tigers striker Fryatt drove home an 81st-minute penalty after he had been fouled by Gunter, while Radoslaw Majewski fired a consolation for Forest in stoppage time.
Hull made just one change with Paul McKenna recalled to midfield in place of the injured Tom Cairney, while defender Greg Cunningham, midfielder Andy Reid and striker Marcus Tudgay returned for Forest.
The visitors were first to threaten in the 12th minute through Paul Anderson, whose header from Reid's cross was wide of the target.
Hull's first effort on goal fell to Australian Richard Garcia, who saw his 25-yard shot comfortably held by Forest goalkeeper Lee Camp.
Hull playmaker Robert Koren hobbled out of the game injured in the 32nd minute and was replaced by Josh King, and although Corry Evans volleyed over the crossbar from 20 yards, clear-cut first-half chances were few and far between.
Mannone held Majewski's shot from inside the area early in the second half and McGugan tested the Italian goalkeeper again with another long-range effort.
Mannone, on loan from Arsenal, pulled off a superb save to keep out Majewski's angled shot in the 53rd minute, but during Forest's best spell Hull took a fortunate lead in the 58th minute.
Rosenior's shot from the right side of the area came back off the post and rebounded off Forest defender Gunter beyond Camp.
Reid vented his frustration by fouling McKenna after the restart and was duly booked by referee Geoff Eltringham, but Forest quickly set about chasing an equaliser.
Guedioura lashed a 20-yard shot over the crossbar, Anderson pulled his angled effort wide and Reid's goalbound shot was blocked by Hull defender James Chester.
Hull then made the game safe in the 80th minute when Gunter was adjudged to have brought down Fryatt in the box, and the Tigers striker picked himself up to power home the subsequent penalty low into the centre of goal.
Gunter went some way to amending his misdemeanours in the 93rd minute when he crossed for Majewski to convert from 12 yards.
