# Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models

arXiv: https://arxiv.org/abs/2206.09917 · HTML: https://mlsanity.com/paper/2206.09917/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 66 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Counterfactual Fairness in Text Classification through Robustness](https://mlsanity.com/paper/1809.10610/) (2018)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)](https://mlsanity.com/paper/2006.07235/) (2020)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [Challenges for Toxic Comment Classification: An In-Depth Error Analysis](https://mlsanity.com/paper/1809.07572/) (2018)
- [Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hate](https://mlsanity.com/paper/2108.05921/) (2021)
- [Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis](https://mlsanity.com/paper/2010.04543/) (2020)
- [Detecting East Asian Prejudice on Social Media](https://mlsanity.com/paper/2005.03909/) (2020)
- [Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXEL](https://mlsanity.com/paper/2004.13850/) (2020)
- [Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages](https://mlsanity.com/paper/2112.09301/) (2021)
- [Exploring Transformer Based Models to Identify Hate Speech and Offensive Content in English and Indo-Aryan Languages](https://mlsanity.com/paper/2111.13974/) (2021)

## Built on by (most cited first, 30 of 51)

- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [CultureLLM: Incorporating Cultural Differences into Large Language Models](https://mlsanity.com/paper/2402.10946/) (2024)
- [Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models](https://mlsanity.com/paper/2401.16727/) (2024)
- [From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets](https://mlsanity.com/paper/2404.17874/) (2024)
- [CulturePark: Boosting Cross-cultural Understanding in Large Language Models](https://mlsanity.com/paper/2405.15145/) (2024)
- [Enhancing LLM Safety Through a Theoretical Minimax Game Lens](https://mlsanity.com/paper/2502.05163/) (2025)
- [MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector](https://mlsanity.com/paper/2401.05060/) (2024)
- [Benchmarking LLM Guardrails in Handling Multilingual Toxicity](https://mlsanity.com/paper/2410.22153/) (2024)
- [Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions](https://mlsanity.com/paper/2502.11073/) (2025)
- [Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset](https://mlsanity.com/paper/2403.19559/) (2024)
- [HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter](https://mlsanity.com/paper/2411.15462/) (2024)
- [AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages](https://mlsanity.com/paper/2501.08284/) (2025)
- [Multilingual Auxiliary Tasks Training: Bridging the Gap between Languages for Zero-Shot Transfer of Hate Speech Detection Models](https://mlsanity.com/paper/2210.13029/) (2022)
- [SGHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Singapore](https://mlsanity.com/paper/2405.01842/) (2024)
- [From Surveys to Narratives: Rethinking Cultural Value Adaptation in LLMs](https://mlsanity.com/paper/2505.16408/) (2025)
- [Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages](https://mlsanity.com/paper/2210.11359/) (2022)
- [DefVerify: Do Hate Speech Models Reflect Their Dataset's Definition?](https://mlsanity.com/paper/2410.15911/) (2024)
- [Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study](https://mlsanity.com/paper/2505.06149/) (2025)
- [Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation](https://mlsanity.com/paper/2505.24754/) (2025)
- [Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detection](https://mlsanity.com/paper/2305.13276/) (2023)
- [Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs](https://mlsanity.com/paper/2601.03087/) (2026)
- [Validating Multimedia Content Moderation Software via Semantic Fusion](https://mlsanity.com/paper/2305.13623/) (2023)
- [IberBench: LLM Evaluation on Iberian Languages](https://mlsanity.com/paper/2504.16921/) (2025)
- [MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch](https://mlsanity.com/paper/2509.12340/) (2025)
- [CoRAL: a Context-aware Croatian Abusive Language Dataset](https://mlsanity.com/paper/2211.06053/) (2022)
- [AustroTox: A Dataset for Target-Based Austrian German Offensive Language Detection](https://mlsanity.com/paper/2406.08080/) (2024)
- [Toxic language detection: a systematic review of Arabic datasets](https://mlsanity.com/paper/2312.07228/) (2023)
- [GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection?](https://mlsanity.com/paper/2402.15238/) (2024)
- [Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignment](https://mlsanity.com/paper/2406.11039/) (2024)
- [Cross-lingual Offensive Language Detection: A Systematic Review of Datasets, Transfer Approaches and Challenges](https://mlsanity.com/paper/2401.09244/) (2024)

## Abstract (arXiv)

Hate speech detection models are typically evaluated on held-out test sets. However, this risks painting an incomplete and potentially misleading picture of model performance because of increasingly well-documented systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, recent research has thus introduced functional tests for hate speech detection models. However, these tests currently only exist for English-language content, which means that they cannot support the development of more effective models in other languages spoken by billions across the world. To help address this issue, we introduce Multilingual HateCheck (MHC), a suite of functional tests for multilingual hate speech detection models. MHC covers 34 functionalities across ten languages, which is more languages than any other hate speech dataset. To illustrate MHC's utility, we train and test a high-performing multilingual hate speech detection model, and reveal critical model weaknesses for monolingual and cross-lingual applications.