# Antisemitic Messages? A Guide to High-Quality Annotation and a Labeled Dataset of Tweets

arXiv: https://arxiv.org/abs/2304.14599 · HTML: https://mlsanity.com/paper/2304.14599/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 6 of 6)

- [Towards generalisable hate speech detection: a review on obstacles and solutions](https://mlsanity.com/paper/2102.08886/) (2021)
- ["Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learning](https://mlsanity.com/paper/2104.05947/) (2021)
- [On the Globalization of the QAnon Conspiracy Theory Through Telegram](https://mlsanity.com/paper/2105.13020/) (2021)
- [Annotating Antisemitic Online Content. Towards an Applicable Definition of Antisemitism](https://mlsanity.com/paper/1910.01214/) (2019)
- [Codes, Patterns and Shapes of Contemporary Online Antisemitism and Conspiracy Narratives -- an Annotation Guide and Labeled German-Language Dataset in the Context of COVID-19](https://mlsanity.com/paper/2210.07934/) (2022)
- [Uncovering the Dark Side of Telegram: Fakes, Clones, Scams, and Conspiracy Movements](https://mlsanity.com/paper/2111.13530/) (2021)

## Built on by (most cited first, 6 of 6)

- [Using LLMs to discover emerging coded antisemitic hate-speech in extremist social media](https://mlsanity.com/paper/2401.10841/) (2024)
- [Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology](https://mlsanity.com/paper/2406.14099/) (2024)
- [Evaluating Large Language Models for Detecting Antisemitism](https://mlsanity.com/paper/2509.18293/) (2025)
- [Cracking the Code: Enhancing Implicit Hate Speech Detection through Coding Classification](https://mlsanity.com/paper/2506.04693/) (2025)
- [Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks](https://mlsanity.com/paper/2604.17022/) (2026)
- [The Surge of Anti-Semitism in German Social Media following the October 7 Attacks](https://mlsanity.com/paper/2609.36290/) (2026)

## Abstract (arXiv)

One of the major challenges in automatic hate speech detection is the lack of datasets that cover a wide range of biased and unbiased messages and that are consistently labeled. We propose a labeling procedure that addresses some of the common weaknesses of labeled datasets. We focus on antisemitic speech on Twitter and create a labeled dataset of 6,941 tweets that cover a wide range of topics common in conversations about Jews, Israel, and antisemitism between January 2019 and December 2021 by drawing from representative samples with relevant keywords. Our annotation process aims to strictly apply a commonly used definition of antisemitism by forcing annotators to specify which part of the definition applies, and by giving them the option to personally disagree with the definition on a case-by-case basis. Labeling tweets that call out antisemitism, report antisemitism, or are otherwise related to antisemitism (such as the Holocaust) but are not actually antisemitic can help reduce false positives in automated detection. The dataset includes 1,250 tweets (18%) that are antisemitic according to the International Holocaust Remembrance Alliance (IHRA) definition of antisemitism. It is important to note, however, that the dataset is not comprehensive. Many topics are still not covered, and it only includes tweets collected from Twitter between January 2019 and December 2021. Additionally, the dataset only includes tweets that were written in English. Despite these limitations, we hope that this is a meaningful contribution to improving the automated detection of antisemitic speech.