Budge took control of Hearts on Friday after the Edinburgh club narrowly avoided liquidation following the free-spending years of Vladimir Romanov which left them mired in debt.
Hearts were relegated to the Championship, having started last season on minus 15 points and with a transfer embargo after entering administration.
On a busy day at Tynecastle, former boss Craig Levein has been appointed as director of football, with Robbie Neilson replacing Gary Locke as head coach.
And Budge insists Hearts will work to a long-term strategy in order to return to the Scottish Premiership with solid foundations, even if it means missing out on immediate promotion.
A statement from Budge read: "It was clear from my earliest visits to Tynecastle and my earliest meetings with senior staff that the company, Heart of Midlothian plc, was, in every sense of the word, "broken".
"One thing is very clear. We have come far too close to losing our very existence and we must never allow this to happen again. We must grasp this opportunity to start afresh; to put in place solid foundations that will ensure we never again risk our 140 years of history.
"We must make changes. We must stop thinking short-term. We must put in place a three-to-five-year plan that will see our commercial activities totally re-vamped, such that they once again provide a solid financial under-pinning to our football club; and we must also completely revitalise and re-focus the football side of our club.
"We have some excellent young players, who have done us proud in the last season.
"We owe it to them to create an environment that will help them to continue to flourish and develop. This will take time, patience and the continued loyal support of Hearts' dedicated fan base. We will implement strategies to grow and strengthen the club. We will live within in our means to ensure that the future of the club is never again put in jeopardy.
"We must ensure we have a clear footballing strategy that drives every aspect of our game; everything from who we recruit into our club, from youth academy through to first team players; how we approach coaching, at all age groups and all levels; how we undertake training... and indeed, what kind of football we play at Hearts.
"There must be a totally cohesive plan that ties all aspects of our club together. Everyone must be pulling in the same direction.
"Next season cannot simply be about getting back up into the SPFL. Of course, we will be striving for that, but we must take a longer-term view and build for the future."