Lewandowski underlined his status as one of the hottest properties in world football on Wednesday night when he scored all four goals in Dortmund's Champions League semi-final thrashing of Real Madrid.
Had it not been for an Icelandic ash cloud, or the financial constraints of the board at Ewood Park - Allardyce cannot remember which - the Pole could today have been fighting relegation to League One with Blackburn instead.
Rovers were one of a number of clubs, including Tottenham and West Brom, who were offered Lewandowski by his then parent club Lech Poznan for just three million euros (£2.5m) in 2010.
"It's true," the West Ham boss said. "He was put forward to sign very strongly by our recruitment staff and unfortunately it was one that got knocked back.
"My distant memory isn't that great but I think (former Rovers chairman) John (Williams) and the board said we are not going to commit that amount of money.
"I think they wanted three or four million euros but we were particularly strapped financially at the time at Blackburn.
"I watched him play. But I didn't get the chance to meet him.
"I think his agent said he couldn't come over because of the ash cloud, yes.
"He's probably worth 40 million euros after last (Wednesday) night."
It goes without saying that if Blackburn did have the man who ran rings around the Real defence in their team, they would not be at the wrong end of the npower Championship now.
And for Allardyce, who spent just under three years in charge of Blackburn, knowing that he missed out on coaching the talented 24-year-old is particularly galling.
"I think about it every time I see him play. Last night I was thinking it again," the 58-year-old said.
"He would have had a period of getting used to the level and the physical presence, but with a talent like that it wouldn't have been too long before he started to shine and score goals.
"The level he is playing at now is unbelievable, not just his scoring ability but his ability to retain possession.
"There were a lot of people looking at him at that time as well but we had the edge over every one else because of the attraction of the Premier League. He could have used us as a stepping stone but it didn't happen."
Manchester United have been linked with a big-money move for the striker, who has one year left to run on his contract, but Allardyce thinks the allure of English game is not as strong as it once was.
"I think we (English football) have had a bit of a knock-back this year," said Allardyce, referring to the lack of an English team in the last four of the Champions League.
"I am not sure if that will do anything to us just yet in terms of our status in the Champions League.
"I think that all those big (English) clubs will realise that they have to go out and invest even more again and hopefully we can grow back to where we were."
West Ham's opponents on Saturday - Manchester City - are tipped to be the biggest spenders in the market this summer despite the introduction of new financial fair play rules which mean clubs have to balance their books.
When asked whether the new rules will hurt last year's champions, Allardyce said: "It depends on the turnover and where they see how big the turnover is.
"How much did they get Etihad Stadium sponsored for? That has increased their revenues. They will probably find that their shirt sponsorship deal will be increased and is larger than before and that increases the turnover."