# Evaluating Large Language Models for Antisemitic Incident Classification

arXiv: https://arxiv.org/abs/2607.04890 · HTML: https://mlsanity.com/paper/2607.04890/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 71 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Builds on (most cited first, 29 of 29)

- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [Towards generalisable hate speech detection: a review on obstacles and solutions](https://mlsanity.com/paper/2102.08886/) (2021)
- [Investigating Reasons for Disagreement in Natural Language Inference](https://mlsanity.com/paper/2209.03392/) (2022)
- [From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models](https://mlsanity.com/paper/2305.17174/) (2023)
- ["Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learning](https://mlsanity.com/paper/2104.05947/) (2021)
- [NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps](https://mlsanity.com/paper/2404.01651/) (2024)
- [Hate Speech Criteria: A Modular Approach to Task-Specific Hate Speech Definitions](https://mlsanity.com/paper/2206.15455/) (2022)
- [Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways](https://mlsanity.com/paper/2406.11980/) (2024)
- [HateBR: A Large Expert Annotated Corpus of Brazilian Instagram Comments for Offensive Language and Hate Speech Detection](https://mlsanity.com/paper/2103.14972/) (2021)
- [GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction](https://mlsanity.com/paper/2405.15760/) (2024)
- [Understanding and Detecting Hateful Content using Contrastive Learning](https://mlsanity.com/paper/2201.08387/) (2022)
- [A Legal Approach to Hate Speech: Operationalizing the EU's Legal Framework against the Expression of Hatred as an NLP Task](https://mlsanity.com/paper/2004.03422/) (2020)
- [Tracing Antisemitic Language Through Diachronic Embedding Projections: France 1789-1914](https://mlsanity.com/paper/1906.01440/) (2019)
- [Leveraging Annotator Disagreement for Text Classification](https://mlsanity.com/paper/2409.17577/) (2024)
- [It's a Thin Line Between Love and Hate: Using the Echo in Modeling Dynamics of Racist Online Communities](https://mlsanity.com/paper/2012.01133/) (2020)
- [Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles](https://mlsanity.com/paper/2406.06840/) (2024)
- [Codes, Patterns and Shapes of Contemporary Online Antisemitism and Conspiracy Narratives -- an Annotation Guide and Labeled German-Language Dataset in the Context of COVID-19](https://mlsanity.com/paper/2210.07934/) (2022)
- [Making FETCH! Happen: Finding Emergent Dog Whistles Through Common Habitats](https://mlsanity.com/paper/2412.12072/) (2024)
- [Factoring Hate Speech: A New Annotation Framework to Study Hate Speech in Social Media](https://mlsanity.com/paper/2311.03969/) (2023)
- [Examining Untempered Social Media: Analyzing Cascades of Polarized Conversations](https://mlsanity.com/paper/1906.04261/) (2019)
- [How toxic is antisemitism? Potentials and limitations of automated toxicity scoring for antisemitic online content](https://mlsanity.com/paper/2310.04465/) (2023)
- [Evaluating Large Language Models for Detecting Antisemitism](https://mlsanity.com/paper/2509.18293/) (2025)
- [Monitoring the evolution of antisemitic discourse on extremist social media using BERT](https://mlsanity.com/paper/2403.05548/) (2024)
- [Leader-driven or Leaderless: How Participation Structure Sustains Engagement and Shapes Narratives in Online Hate Communities](https://mlsanity.com/paper/2512.12441/) (2025)

## Abstract (arXiv)

Addressing hate and violence in society requires timely detection of hateful events from public reporting, but automated identification of hateful events remains underexplored. We introduce the task of hateful event detection and investigate the ability of AI systems, specifically large language models (LLMs), to discover and classify reports of antisemitic events with fine-grained labels. We evaluate OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on multiple expert-annotated datasets containing antisemitic event descriptions from news articles, civil society reports, and official records. We show that LLMs, particularly GPT-4o, have potential for this task, but substantial improvement is needed. Providing clear term definitions and in-context examples in prompts can improve performance: definitions are most helpful for rhetoric-oriented events (e.g. classical antisemitic tropes), while examples help label action-oriented events (e.g. physical assault). A case study of college newspapers demonstrates that LLMs can help surface relevant real-world events, supporting early monitoring and intervention. Overall, our findings highlight both opportunities and critical gaps in AI's ability to recognize complex harms and underscore the need for collaborative efforts among AI developers, policymakers, and civil society to design models, implement robust evaluation, and develop policy frameworks for defining and combating hate efficiently and effectively.