The city council elected as Mayor Herman Mashaba from the opposition Democratic Alliance (DA).
The ANC had run the city since the fall of apartheid more than 20 years ago. It lost its council majority in local elections, although it is still the largest party.
The party has also lost control of the capital Pretoria and Cape Town.
It follows local elections earlier this month that produced no outright winner in most districts, resulting in hung municipalities and forcing parties to form coalitions to govern.
Of the country’s six biggest cities, the ANC only won an outright majority in Durban, seen as a stronghold for South African President Jacob Zuma.
There was drama at Monday’s Johannesburg council meeting, which lasted 11 hours.
A scuffle broke out between opposition party members and electoral commission officials and an ANC councillor who was sworn in earlier in the day collapsed and died shortly after Mr Mashaba was elected.
The ANC had won 44.5% of the vote, more than the DA’s 38.4%. But the left-wing Economic Freedom Fighters (EFF) with 11% found itself in the position of kingmaker and refused to give its votes to the ANC.
The election of the Democratic Alliance’s Herman Mashaba as mayor of Johannesburg represents a tectonic shift in South African politics.
This is the first time since the end of apartheid that the ANC lost control of the city of gold, as Johannesburg is known locally.
The Johannesburg wing of the ANC is known within the party to be professional and not corrupt.
However, voters still punished it for the litany of corruption scandals the ANC has been involved in at a national level.
And there were some local problems, including potholed roads, a shambolic electricity billing system and refuse strikes that left the city filthy.