The International Criminal Tribunal for the former Yugoslavia also said that Serbia failed to punish those who carried out the Srebrenica massacre.
Bosnia brought the case against Serbia last year, and has sought billions of dollars from Serbia in compensation.
But the court rejected Bosnia's claim for reparations.
"Financial compensation is not the appropriate form of reparation," the ruling said.
The case is the first against a state to brought before the war crimes tribunal in The Hague.
The Srebrenica massacre is the only event in the Bosnian war that the ICJ has established as genocide.
The judges have already handed down two individual genocide convictions against Bosnian Serb officers responsible for what is described as Europe's worst massacre of civilians since World War II.
About 8,106 Bosnian Muslims were murdered in 1995 after Bosnian Serb forces seized Srebrenica, a UN-protected enclave.
Commenting on the ruling, the Bosnian Muslim member of the country's tripartite presidency, Haris Siladzic, told the BBC there was "disappointment" among Bosnian Muslims.
However he said he was pleased that the court had "ruled that Serbia and Montenegro had violated the Convention on the Prevention and Punishment of the Crime of Genocide by not preventing or punishing the perpetrators of the genocide".
Bosnia had accused Serbia of masterminding the widespread "ethnic cleansing" during the brutal 1992-95 war that left over 200,000 Bosnian Muslims and Croats dead.
While the ICJ ruled that Belgrade had given "considerable military and financial support" to the Bosnian Serb leadership, the judges said they found no evidence to suggest that Serbia was directly responsible for the Srebrenica massacre.
However, under the 1948 genocide convention Belgrade was also obliged to do everything within its power to prevent genocide especially because it did have influence over the Bosnian Serb leadership, the court said.
The ICJ also called on Serbia to fully cooperate with the trial and deliver Bosnian Serb general Ratko Mladic, who is indicted for genocide over the Srebrenica massacre, to the war crimes court along with other suspects.
Güncelleme Tarihi: 20 Eylül 2018, 18:16