The winger set up United's first three goals before producing a wonder strike to seal the victory.
Scott Severin scored his first goal for United after 25 minutes with a downward header before David Goodwillie nodded home seven minutes before the break.
Goodwillie doubled his tally when he headed home two minutes after the restart before Conway netted with a wonderful strike after 49 minutes.
Substitute David Silva pulled a goal back for Killie with a long-range strike before Manuel Pascali nodded home a second in the dying stages, but United held on for a deserved win.
The result moves United eight points clear of fifth-placed Kilmarnock, who have now gone four matches without a win under caretaker boss Kenny Shiels.
United were without Morgaro Gomis, Sean Dillon and Mihael Kovacevic who were all shown red cards in their heavy midweek defeat to Rangers.
Keith Watson, Barry Douglas and Prince Buaben came in to replace the suspended trio, while Johnny Russell made way for Conway.
Kilmarnock made two changes from the team that conceded four at home to Celtic with goalkeeper Cameron Bell replacing Anssi Jaakkola and Garry Hay starting ahead of Silva who dropped to the bench.
United had won seven of their previous nine home matches in the SPL and started the game brightly with Buaben flashing a shot over the bar from distance and Scott Robertson curling an effort wide.
The hosts went desperately close when Goodwillie robbed Pascali of possession before crashing a thunderous drive off the underside off the crossbar.
Bell produced a fine stop to deny Danny Swanson before Pascali picked up the first booking of the match for a foul on United skipper Garry Kenneth.
United were thwarted by the woodwork for a second time from the resulting set-piece with Douglas' free-kick bouncing back off the bar.
And the inevitable opener for United came after 25 minutes when a Conway corner was headed home by the unmarked Severin.
The Terrors almost gifted Killie an equaliser when William Gros raced onto Buaben's careless pass, but the Frenchman's shot was well stopped by Dusan Pernis.
However, United scored a deserved second after 38 minutes with Conway again providing the ammunition with an enticing delivery into the penalty area which was bravely headed home at the near post by Goodwillie.
Killie made a double switch at the break with Silva and James Dayton introduced at the expense of skipper Craig Bryson and Ryan O'Leary.
However, it was United who extended their lead within two minutes of the restart when Conway produced another teasing cross for Goodwillie, who headed past the despairing Bell for his 16th league goal of the season.
And the Kilmarnock goalkeeper was picking the ball out of his net again after 49 minutes when Conway unleashed a magnificent strike from 25 yards that flew into the top corner.
The visitors pulled a goal back after 70 minutes thanks to substitute Silva's equally stunning strike from distance.
Conway rifled another effort over the bar with his left foot as the hosts searched for a fifth.
But it was Kilmarnock who scored a second with three minutes remaining when Pascali rose to head home Silva's in-swinging corner.