West Ham will become anchor tenants of the stadium - the centrepiece for London 2012 - from August 2016 after being awarded sole use of the venue by the London Legacy Development Corporation.
Leyton Orient wanted to share the venue with West Ham but lost their bid to win a judicial review into the decision in September.
The House of Lords Olympic and Paralympic Legacy Committee was appointed to look into the impact of the Games in London, particularly regarding regeneration and sporting development, and although it did not specifically re-examine the awarding of the Olympic Stadium to West Ham, Lord Harris did have some strong opinions on the situation.
He told Sky Sports News: "We've not tried to re-visit the process. What we are saying is that the rather unedifying spectacle of West Ham and Leyton Orient fighting among themselves like two school children in the playground arguing about who can go down the slide first is not really helpful.
"The Olympic Stadium is a national asset. We want to make sure that asset is used to the full and that the community at large gets the widest possible benefit."
When asked if that could potentially mean Leyton Orient ground sharing or playing some of their games at the Olympic Stadium, he replied: "We want to make sure all the different parties sit down and talk properly. We think it may be possible to work in a different way.
"It has to be economically viable. You can't open up a huge stadium if there aren't going to be enough people going into it to make it worthwhile. But the most effective use of that stadium has to be in the national interests. It is a national asset - let's make sure it is used as a national asset."
The committee's report into the legacy of the 2012 Olympic and Paralympic Games has now been published and Lord Harris has revealed there has been no major increase in sporting participation in the country - one of the key aims - since the Games finished.
He said: "The London Games were probably the most successful ever in terms of planning for a legacy. We could be achieving even more and we have to work hard to continue to meet the aspirations that were set in the first place.
"Clearly there was a hope in there was going to be a step change in sporting participation. In the event, the long-term general improvement in sports participation has continued but there is no evidence of a step change in the last year.
"If anything, figures have gone down. We can only continue to achieve the long-term improvement in sports participation by sustained leadership and by making sure there is investment in facilities and proper investment in physical education.
"I don't think the aspiration to improve sporting participation has yet been delivered. It is still possible. It needs continued investment - both in terms of physical education of school-age children and also in terms of the improvements of local sporting facilities.
"It is undoubtedly the case that more should have been done more quickly to capitalise on the enthusiasm. A lot of that momentum has been lost. Is it too late? Well, we are embarking on a "decade of sport". Let's see what we can do to make sure the "flame is kept alive"."
The committee has recommended two key changes going forward.
Lord Harris added: "First, it has to be made explicit which cabinet minister has got responsibility taking forward all the aspects of the Olympic legacy. There should be proper focus from the Government on that.
"Secondly, as far as London and east London is concerned, the Mayor of London's office should take on the responsibility of the regeneration of the area and be responsible for delivering that regeneration."
Lord Harris signed off by rating the legacy. "Good effort - could do an awful lot better," he said.