Offering words of encouragement for New Synagogue rabbi Gesa Ederberg on a Yom Kippur holiday tainted by violence, Merkel said: "Unfortunately, on your holy day today, we've witnessed something horrible. Two people have been killed, and there has been an attack on Jews in Germany."
"My aim, and that of all politicians, is to do everything to ensure you can live safely. And this day shows that it hasn't been enough, that we have to do more," she added.
Two people were shot dead, and two others injured, in the German city of Halle earlier Wednesday in an anti-Semitic attack as Jews marked the holy day of Yom Kippur.
Rabbi Ederberg said she had received many messages from people who were "afraid".
"Many people told me they weren't coming because they were scared. That has to change," she added.
The rabbi blasted a "resurgence of anti-Semitism in Germany," urging "civil society to oppose anyone who would use the term 'Jew' as an insult and to battle "all brown (fascistic) forces".
Maintaining a ceremonial silence under the light rain, those present placed candles in front of the synagogue.
Several people were carrying Israeli flags on their shoulders. Others unrolled a banner reading "against all anti-Semitism".
"It was important to be here to show our solidarity with the Jewish community. An attack on Jews in Germany is an attack on all of us," said event organiser Sawsan Chebli, a Berlin city representative of Palestinian origin.
"Something we thought impossible just a few years ago has reappeared. It's up to us to stay vigilant," said Verena Schoenwaelder, an atheist Berliner who attended with her young daughter.
"We show solidarity," said imam Kadir Sanci.
"When someone is attacked, you have to support them. Today it's the Jews who are the victims, tomorrow it will be Muslims. Either way, let's stick together."
Away from Berlin, several hundred people spontaneously gathered at the marketplace in Halle, placing candles and flowers on the ground.
Interior Minister Horst Seehofer labelled the Halle killings an "anti-Semitic attack", likely carried out by a "far-right" perpetrator.
The shooting sparked a lockdown in the city, before the suspect was captured.
During his deadly rampage, the man had attempted to storm the synagogue in Halle with a shotgun and an automatic weapon but failed to breach its doors.
Shocked but unharmed, the 80 worshippers inside were later transferred to the city hospital, where they finished their prayers and broke their fast before being taken to a safe location, AFP learned.