Israeli radio reported considerable damage to the Zion cemetery on Wednesday, with gravestones damaged and crosses broken.
The Jerusalem Post, citing Jerusalem University College, reported that the perpetrators were two Jewish teenagers, saying the damage amounted to the equivalent of $99,700.
An Israeli police spokesman confirmed on Wednesday that an investigation was under way, but did not comment on the identity of the suspects.
A video shared on social media showed two figures throwing large stones at graves and overturning a cross.
The incident reportedly occurred on Sunday.
The cemetery, founded in 1848, also serves as a burial place for Jerusalem's German-speaking Protestant community. The German ambassador to Israel, Steffen Seibert, spoke on Twitter of "contemptible criminal behaviour" by the perpetrators. "I hope the police will investigate, find these guys and bring them to justice."
The desecrated graves included those of three Palestinian police officers and members of various Protestant congregations, according to media reports, including bishop Samuel Gobat, Jerusalem's second Protestant bishop. He had bought the land for the cemetery.
The cemetery has been desecrated before and there have been previous incidents of hostility towards Christians in Jerusalem.
