The Australian government has now begun cancelling the passports of a number of Hajj pilgrims in a report in the Sydney Morning Herald.

The Department of Immigration and Border Protection distributed a note to Hajj pilgrims in August, explaining that no one would be stopped on the basis of their race, religion or destination but "they need to be aware that the Australian Border Force is committed to maintaining the security of our nation and will screen all arriving and departing travellers".

Omar Chandab, 34, had first started his planning his first religious pilgrimage, the Hajj, with his wife seven years ago. He renewed his passport two months ago and arranged for his four children to be looked after relatives.

At the airport, he and his wife were  pulled aside at Melbourne airport on Saturday and questioned separately for two hours by the feseral police about whether they supported any overseas conflicts or political groups.

It was then that were told they wouldn't be flying that day and were sent home. On Monday, Mr Chandab received a letter saying his passport had been cancelled. 

The father of four who has a successful construction business and coaches several sport teams has vowed to mobilise the Muslim community in a fight against the government's counter-terrorism strategy. 

"I've got nothing to hide so I'm taking it to the top," said Mr Chandab, who has engaged lawyer Charlie Atlas to launch a constitutional challenge in the High Court.

"In these times, the last thing you want is for innocent people to feel that they are being isolated or don't deserve the right to travel.

"What will the average Muslim think of that? If he is helpless, you think he won't turn into an extremist?"

Fairfax Media has seen ASIO's negative security assessment of Mr Chandab, which is based on two interviews from three years ago.

The spy agency contacted him in November and December, 2012, because he knew two men, Roger Abbas and Sammy Salma, through kick-boxing.

Both these men had travelled  in the early days of the civil war in 2012 and were killed while fighting Assad forces.

Mr Chandab was interviewed by ASIO and he then contacted them again soon after to provide more information, the assessment said.

The calls are now the basis for ASIO's assessment that he "likely adheres to an extremist ideology" and "continues to associate with individuals" like Abbas and Salma.

"ASIO assesses that Mr Chandab would be likely to engage in politically-motivated violence... if he were to continue to hold an Australian passport," it said.

It's not clear why Mr Chandab was assessed as suitable for a passport two months ago but not on Saturday, considering the intelligence relied upon was three years old.

Mr Chandab had said he has never seen such behaviour by the country he was born in. 

"I am absolutely speechless, I'm astonished. They think I'd leave my four beautiful kids, my successful construction business, my professional football team? They're just going on random assumptions that I want to engage in violence" he said.

Mr Atlas said other Muslims have also been targeted by ASIO for being informants, when "in reality... [they] have no connection to illegal activity or extremists".

"In an attempt to curb radicalisation and extremism the policies are achieving the opposite of what is intended," he said.

Forty-five passports were cancelled on security grounds in 2013-14, more than double the previous year.

All agencies involved said they do not comment on individual cases.