Many of those displaced by the fighting between Azeri and ethnic Armenian forces in the 1990s face discrimination and poverty, Amnesty says in a report.
It estimates more than 600,000 Azeris remain isolated despite a ceasefire.
The energy-rich former Soviet republic has one of the world's worst problems with internally-displaced people.
More than 600,000 Azeris have not been able to return to their homes since the ceasefire in Nagorno-Karabakh in 1994, the report says.
They remain a cause of huge concern for a relatively poor country with a population of only eight million, despite its booming oil economy.
Some still live in run-down collective housing or refugee camps, others in newly-built settlements in remote locations where work is hard to find.
Amnesty International says this has caused them to become isolated and segregated, and to suffer more from poverty and illness.
Earlier this year, the UN praised the Azeri government's efforts to address some of these problems by closing some of the worst tent camps but it also said more should be done to improve living conditions.
Azerbaijan effectively lost the war with Armenian forces over Nagorno-Karabakh.
The Azeri government insists it will eventually regain control over the territory, although little progress has been made in peace negotiations in more than a decade.
The long-term impact of the conflict also continues to affect neighbouring Armenia.
Some 400,000 Armenians fled their homes during the war. Many of them still live in desperate conditions, just like their counterparts in Azerbaijan.
Last Mod: 28 Haziran 2007, 15:01