About 3,000 Israeli police had been deployed amid continuing anger over Israeli archaeological and repair work near the mosque compound, which opponents claim endanger its foundations.
Adnan al-Husseini, director of the Muslim authority which runs the compound, said 2,500 worshippers attended prayers despite Israeli restrictions, although police put the figure at 6,000.
Tens of thousands of Muslims normally attend Friday prayers at al-Aqsa mosque.
Güncelleme Tarihi: 20 Eylül 2018, 18:16
Preachers condemned the Israeli work and Yussef Abu Snieneh, a religious leader, said it posed a "great danger to our mosque".
He accused police of transforming the revered compound into a "battleground" last Friday, when they stormed it amid clashes with Palestinian stonethrowers.
At least 20 Palestinians and 15 Israeli police were injured in the violence.
Only Muslim men aged over 50 and in possession of Israeli identity cards were granted entrance to the main weekly service, compared to those over 45 last week.
"It feels like we're coming to a battlefield and not a holy site," said al-Husseini.
The excavations, which started on February 6, have angered Muslims across the world and provoked violent protests.
Israel denies the work poses any risk to the holy site, also venerated as the location of the ancient Jewish temple destroyed by the Romans in 70AD.
Sheikh Raed Salah, the head of Israel's Islamic Movement, banned by an Israeli court from approaching the Old City, led his own prayers in east Jerusalem and delivered a speech before hundreds of supporters.
Isolated clashes broke out after the prayers between Israeli police, who used stun grenades and water cannons on protesters.
The authorities say the excavations are a vital prerequisite to construction work to replace a damaged wooden ramp leading to the compound with a stone bridge, work that the Jerusalem mayor has suspended because of Arab protests.
Israel has agreed to allow Turkish officials to inspect the work, following talks between the Israeli and Turkish prime ministers.
Israel is also broadcasting a live webcast of the excavation site 24 hours a day in a further bid to appease Muslim anger.