Judges rejected a challenge by the unvaccinated tennis star after the government cancelled his visa on "health and good order" grounds.
Djokovic said he was "extremely disappointed" but accepted the ruling. He has left on a flight to Dubai.
It marks the end of a 10-day saga, in which the Serb fought to stay to defend his title in the Australian Open.
Djokovic's supporters fell silent outside the courtroom as the decision was announced on the eve of what would have been his opening match in the tournament. One fan told the BBC her summer would be "empty" without the 34-year-old playing at the Open.
Australian Prime Minister Scott Morrison welcomed "the decision to keep our borders strong and keep Australians safe" but his government faces criticism at home and abroad for its handling of the affair.
Djokovic launched his case after Immigration Minister Alex Hawke used his powers to cancel his visa, arguing his presence in the country risked fanning anti-vaccine sentiment.
During Sunday's court hearing before a three-judge panel, Djokovic's defence unsuccessfully argued that the grounds given by the government were illogical because to deport the star also risked fanning anti-vaccine sentiment.
Chief Justice James Allsop said the court ruling was based on the legality of the minister's decision, not on whether it was the right decision to make.
He promised to release the full reasoning for the ruling in the coming days.
There has been much public anger in Australia over the player's attempt to enter the country without being vaccinated against Covid-19. The federal government has repeatedly said people must comply with the strict laws in place to deal with the pandemic, and that no-one is "above the law".
Djokovic was originally granted a medical exemption to enter Australia by two different independent health panels - one commissioned by Tennis Australia, the other by the state government of Victoria - after testing positive for coronavirus in mid-December.
However, the Australian Border Force detained him on 5 January for not meeting federal coronavirus requirements, and his visa was revoked.
A judge overturned that decision but the government stepped in last Friday to revoke the visa again, saying it was in the public interest.
Although Djokovic is not vaccinated against Covid-19, he has not actively promoted anti-vax disinformation. However, Australian anti-vaxxers have been using the hashtag #IStandWithDjokovic on social media.
Mr Morrison said the centre-right government was "prepared to take the decisions and actions necessary to protect the integrity" of the country's borders.
But Kristina Keneally, a senator for the opposition Labor Party, said Mr Morrison had made himself a "laughing stock on the world stage" by mishandling the Djokovic case.
The government's "litany of failures", she tweeted, had "undermined Australia's border security settings, & provided a lightning rod for the anti-vaccination movement".
A former Labor prime minister, Kevin Rudd, tweeted that the "political circus" could have been avoided had a visa not been issued in the first place.
Serbian President Aleksandar Vucic accused the Morrison government of conducting a "witch hunt".
"He [Djokovic] came to Australia with a medical exemption proposal and then you were mistreating him for 10 days," he said. "Why did you do it? Doing a witch hunt against him? This is something that no one can understand."
The Australian Open could have seen Djokovic make history by winning his 21st Grand Slam. Italy's Salvatore Caruso, ranked 150th in the world, is the "lucky loser" who will now replace Djokovic in his match against Serbia's Miomir Kecmanovic on Monday.
Djokovic said he was "uncomfortable" with the focus placed on him as a result of the the visa row, adding: "I hope that we can all now focus on the game and tournament I love."
He said he was "taking some time to rest and to recuperate" before commenting further.
Deportation orders usually include a three-year ban on returning to Australia, though this can be waived in certain circumstances.
The episode raises further questions for Djokovic, plus other tennis players who are reluctant to be vaccinated against Covid-19, for the season ahead.
Any player eyeing tournaments in America, for instance, will be aware that non-US citizens must be fully vaccinated to travel to the country "with only limited exceptions".
Former Swedish star Mats Wilander told Eurosport that Djokovic's career was "on the line and he might have to do something that he doesn't really want to do".
The men's tennis governing body ATP called the saga a "deeply regrettable series of events", while British tennis star Andy Murray said the situation was "not good" for anyone.