The organisation says many of those returning to Somalia are motivated by fear of being forced out.
Tens of thousands of refugees have been making the journey back home after years in the camp.
Both Kenya and Somalia say it is time Dadaab was shut down.
But HRW says Kenya is not giving the refugees a real choice between being repatriated or staying.
It also says the UN’s refugee arm, the UNHCR, is not giving the refugees accurate information about security conditions in Somalia and they face serious risk of persecution or threats to their lives if they return.
Pressuring refugees to return to a place where their lives or freedom could be under threat is a breach of the 1951 Refugee Convention, the organisation says.
Some of the refugees are also choosing to leave because they fear being deported later and forfeiting a $400 (£300) UN cash grant, the organisation says.
“There is no way these returns can be considered voluntary,” said HRW’s Bill Frelick.
Some refugees who have left Dadaab have since found themselves stranded near the border after regional leaders in Jubaland, which borders Kenya and where most of the returnees are going, stopped receiving them, citing inadequate humanitarian support.
The Dadaab camp in Kenya is home to more than 300,000 Somalis.
It was set up in 1991 to house families fleeing conflict and some people have been living there for more than 20 years.
Kenya says attacks on its soil have been planned in the camp. -BBC