“The Ouagadougou [convention] centre is in our hands,” a spokesman for forces loyal to Libya’s UN-backed government of national accord said.
Sirte, on the Mediterranean coast, is IS’s most significant stronghold in Libya.
The city has been under the militant group’s control since February 2015.
The pro-government forces said IS fighters still held positions in three residential areas of Sirte and in a villa complex near the seafront.
But capturing the Ouagadougou centre had been a key aim of the offensive against IS in the past few weeks.
Fighters also took Sirte’s Ibn Sina hospital and university campus, spokesman Rida Issa said.
At least three fighters were killed and several others wounded, he said. IS was said to have lost at least 20 of its men.
The US has estimated that IS has fewer than 1,000 fighters in Sirte, many of whom are foreigners and veterans of the conflict in Syria and Iraq.
Earlier, a news agency affiliated with IS said the group had brought down a Libyan fighter jet in Sirte.
Pro-government forces said they had lost contact with one aircraft. Two people were reportedly on board.