Robinson's early effort and Atkinson's first Tranmere goal gave Rovers victory and sent the Football League newcomers out.
The home side took a second-minute lead when skipper Robinson headed in from close-range from Ryan Lowe's flick-on.
Tranmere's pressure paid off when Atkinson latched onto Robinson's through-ball in first-half stoppage time and fired home.
Rovers could have doubled their lead earlier when Ash Taylor met Robinson's 13th-minute right-wing corner but the defender headed wide from close-range.
After the break, Mansfield could have got one back when Louis Briscoe received Sam Clucas' cutback in the area but he shot wide just after the hour mark.
Stags midfielder Lindon Meikle volleyed straight at Owain fon Williams two minutes later, and the misses cost the League Two side dearly as they crashed out.