The fast-tracked ceremony comes after his predecessor Pierre Nkurunziza died suddenly last week.
Mr Ndayishimiye is a former rebel leader, like Mr Nkurunziza.
He was backed by his predecessor and was declared the winner of May's presidential election, which the opposition said was rigged.
Mr Nkurunziza died aged 55 on 8 June after suffering a cardiac arrest, the government said. There are unconfirmed reports that he was suffering from Covid-19.
After 15 years in power, Mr Nkurunziza was due to step down in August.
'Fighting genocide'
People attending the inauguration were told to arrive early to allow time for temperature checks and other coronavirus measures - but once in the stadium there was no social distancing
"I will not fail the unity charter, the constitution and other laws, will uphold unity among Burundians, peace and justice for all, [and] fight the ideology of genocide and discrimination," Mr Ndayishimiye said while taking his oath, reports the Reuters news agency.
According to Burundi's constitution, if the head of state dies in office then the president of the national assembly, currently Pascal Nyabenda, should succeed him.
But after a ruling by Burundi's Constitutional Court, Mr Ndayishimiye's inauguration in Gitega was brought forward.
The new president is taking over a country that is diplomatically isolated and on bad terms with donors.
In 2015, the constitution was changed, allowing Mr Nkurunziza to seek a third term, triggering huge protests, an attempted coup and a widespread crackdown.
Thousands of Burundians have fled the country and into refugee camps in neighbouring countries.
Source BBC
