Neil Harris had earlier handed the in-form Lions the lead in Saturday's League One clash and in doing so, equalled Teddy Sheringham's club-record haul of 111 goals for the club.
Alex Nicholls hauled the sides level at Bescot before Frampton headed home the decisive goal in the last minute of normal time.
It was Harris who opened the scoring after 16 minutes, turning quickly to fire under goalkeeper Clayton Ince from a tight angle. Millwall should have doubled their lead six minutes later, but Chris Hackett, clean through on goal, lobbed straight into the hands of Ince.
Hackett's profligacy proved costly when Nicholls volleyed home an equaliser when Michael Ricketts flicked on Chris Palmer's corner.
The visitors always looked the more likely to score a winner and it duly arrived when Karl Moore's corner caused a goalmouth scramble in which Frampton darted in to head in from close range.