This week's battles between Hindus and Muslims have seen mobs armed with swords, guns and acid raze a northeastern district of the Indian capital.
The clashes, which also left almost 200 injured, were triggered by protests against a citizenship law seen by many critics as anti-Muslim and part of Modi's Hindu nationalist agenda.
They exploded into brutal violence on Monday and Tuesday, with residents forced to flee their homes after seeing their homes destroted and a mosque attacked.
"I appeal to my sisters and brothers of Delhi to maintain peace and brotherhood at all times. It is important... calm and normalcy is restored at the earliest," Modi tweeted on Wednesday.
Delhi's chief minister, Arvind Kejriwal, a political opponent, called for the army to be deployed and for a curfew to be imposed over the flashpoint areas.
"Police, despite all its efforts, (are) unable to control situation and instil confidence," Kejriwal tweeted.
- Fear, anger -
Sunil Kumar, the director of the hospital where many of the wounded were taken, told AFP on Wednesday that almost 60 had gunshot injuries.
On Wednesday morning residents cleaned out the blackened interior of a trashed mosque, including a charred Koran, burned out during the violence in the Ashok Nagar area.
A video circulated on social media and verified by AFP showed men ripping off the muezzin's loudspeaker on top of the mosque's minaret and installing a Hindu religious flag.
Locals accused the police of doing nothing to help -- or worse.
"We tried to make many calls to the police... that people are entering our neighbourhoods chanting 'Jai Shree Ram'," said Naeem Malik, referring to a popular Hindu chant.
"But police did not help us at all. We tried to save the women at the protest site but instead policemen started beating us up," Malik said, showing wounds on his leg and hands.
Elsewhere a fire engine tried to put out blazes from the previous night, the air thick with smoke from still-smouldering cars, motorbikes, shops and homes.
"They say we are not Indians but we are Indians by blood," Farhat, 22, a student in Islamic studies, said in her father's shop as police looked on.
"We are afraid, we left our homes. There is no police in the streets at night, just during the day."
The area is home to mostly poorer economic migrants living in many shanty neighbourhoods, and some fled on Wednesday ahead of more expected clashes.
"It is better to leave than to stick around here. Why would we want to die here?," a tailor told AFP, adding that he was returning home to his village in northern India.
- 'Politics of hate' -
The unrest comes amid growing concerns at home and abroad about the direction of India and the future of its 200 million Muslims since Modi's Hindu nationalist BJP swept to a second term last year.
Sonia Gandhi, president of the opposition Congress party, on Wednesday accused BJP figures of giving "inflammatory speeches spreading an atmosphere of hatred and fear", including in Delhi city elections this month.
Congress "appeals to the people to reject the politics of hate," Gandhi said, calling Home Minister Amit Shah, Modi's close ally, "responsible" for the riots.
Since winning a second term, Modi's government has revoked the partial autonomy of Kashmir, India's only Muslim-majority state, and said it wants to conduct a nationwide citizens' register to weed out "infiltrators".
Together with the citizenship law, which fast-tracks claims for persecuted non-Muslim religious immigrants, this has stoked fears that Modi's master plan is to remould India as a Hindu nation, something he denies.
The citizenship law has sparked months of nationwide protests as well as clashes that killed more than 25 people in December.
In recent weeks sit-ins, mostly by women, have sprung up.
The flare-up in violence occurred as US President Donald Trump visited India and held talks with Modi in Delhi on Tuesday. But Trump left as scheduled on Tuesday and his visit was not visibly interrupted by the violence.
bb-ash-ja-abh/stu/kma
20 people have died in sectarian violence in the Indian capital
Mobs were armed with swords, guns and acid
The unrest in Delhi comes amid growing concerns at home and abroad about India's 200 million Muslims since Modi's Hindu nationalist BJP swept to a second term last year
Sonia Gandhi, president of the opposition Congress party, has accused Modi's BJP party leaders of inflaming religious tensions
Map of India showing New Delhi where at least 20 people have been killed after several days of rioting
AFP/File / Eliano IMPERATO 
AFP/File / THOMAS WATKINS 
POOL/AFP/File / Andrew CABALLERO-REYNOLDS 
AFP / Sabrina BLANCHARD 
AFP/File / Money SHARMA 
AFP/File / THOMAS WATKINS 
AFP / John SAEKI 
AFP / ANDREA PATTARO 
AFP / STR 
AFP / JEAN-PHILIPPE KSIAZEK