mlsanityData to 14 Oct 2026

Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models

arXiv 2206.09917 ↗First posted 20 Jun 2022cs.CLcited by 51

Not on the map. Why?

Abstract (arXiv)

Hate speech detection models are typically evaluated on held-out test sets. However, this risks painting an incomplete and potentially misleading picture of model performance because of increasingly well-documented systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, recent research has thus introduced functional tests for hate speech detection models. However, these tests currently only exist for English-language content, which means that they cannot support the development of more effective models in other languages spoken by billions across the world. To help address this issue, we introduce Multilingual HateCheck (MHC), a suite of functional tests for multilingual hate speech detection models. MHC covers 34 functionalities across ten languages, which is more languages than any other hate speech dataset. To illustrate MHC's utility, we train and test a high-performing multilingual hate speech detection model, and reveal critical model weaknesses for monolingual and cross-lingual applications.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 24 of its 66 references that are in the map. First posted on arXiv 20 Jun 2022.

Builds on

24 of 66 references.

References · 24

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  3. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  4. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
  5. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 271
  6. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  7. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  8. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
  9. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
  10. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 209
  11. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  12. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 151
12 more
  1. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  2. 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 140
  3. 2020SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)cited by 138
  4. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  5. 2019Multilingual and Multi-Aspect Hate Speech Analysiscited by 133
  6. 2018Challenges for Toxic Comment Classification: An In-Depth Error Analysiscited by 56
  7. 2021Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hatecited by 50
  8. 2020Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysiscited by 45
  9. 2020Detecting East Asian Prejudice on Social Mediacited by 40
  10. 2020Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXELcited by 23
  11. 2021Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languagescited by 17
  12. 2021Exploring Transformer Based Models to Identify Hate Speech and Offensive Content in English and Indo-Aryan Languagescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 19; 2025: 11; 2026: 9.

51 papers.

Sort:

By area:

  1. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  2. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 50
  3. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  4. 2024From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasetscited by 25
  5. 2024CulturePark: Boosting Cross-cultural Understanding in Large Language Modelscited by 24
  6. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 19
  7. 2024MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detectorcited by 15
  8. 2024Benchmarking LLM Guardrails in Handling Multilingual Toxicitycited by 14
  9. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisionscited by 14
  10. 2024Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Datasetcited by 13
  11. 2024HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twittercited by 13
  12. 2025AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languagescited by 11
38 more
  1. 2022Multilingual Auxiliary Tasks Training: Bridging the Gap between Languages for Zero-Shot Transfer of Hate Speech Detection Modelscited by 10
  2. 2024SGHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Singaporecited by 10
  3. 2025From Surveys to Narratives: Rethinking Cultural Value Adaptation in LLMscited by 9
  4. 2022Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languagescited by 7
  5. 2024DefVerify: Do Hate Speech Models Reflect Their Dataset's Definition?cited by 7
  6. 2025Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Studycited by 7
  7. 2025Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformationcited by 7
  8. 2023Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detectioncited by 6
  9. 2026Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMscited by 6
  10. 2023Validating Multimedia Content Moderation Software via Semantic Fusioncited by 5
  11. 2025IberBench: LLM Evaluation on Iberian Languagescited by 5
  12. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutchcited by 5
  13. 2022CoRAL: a Context-aware Croatian Abusive Language Datasetcited by 4
  14. 2024AustroTox: A Dataset for Target-Based Austrian German Offensive Language Detectioncited by 4
  15. 2023Toxic language detection: a systematic review of Arabic datasetscited by 3
  16. 2024GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection?cited by 3
  17. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  18. 2024Cross-lingual Offensive Language Detection: A Systematic Review of Datasets, Transfer Approaches and Challengescited by 2
  19. 2024JailbreakHunter: A Visual Analytics Approach for Jailbreak Prompts Discovery from Large-Scale Human-LLM Conversational Datasetscited by 2
  20. 2024A Federated Approach to Few-Shot Hate Speech Detection for Marginalized Communitiescited by 2
  21. 2025Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languagescited by 2
  22. 2023Sociocultural knowledge is needed for selection of shots in hate speech detection taskscited by 1
  23. 2023Evaluating the Effectiveness of Natural Language Inference for Hate Speech Detection in Languages with Limited Labeled Datacited by 1
  24. 2023Politeness Stereotypes and Attack Vectors: Gender Stereotypes in Japanese and Korean Language Modelscited by 1
  25. 2024Alternative Speech: Complementary Method to Counter-Narrative for Better Discoursecited by 1
  26. 2026Multilingual Hate Speech Detection and Counterspeech Generation: A Comprehensive Survey and Practical Guidecited by 1
  27. 2023Automated Sentiment and Hate Speech Analysis of Facebook Data by Employing Multilingual Transformer Modelscited by 0
  28. 2023Cordyceps@LT-EDI: Patching Language-Specific Homophobia/Transphobia Classifiers with a Multilingual Understandingcited by 0
  29. 2024AggregHate: An Efficient Aggregative Approach for the Detection of Hatemongers on Social Platformscited by 0
  30. 2024Disentangling Hate Across Target Identitiescited by 0
  31. 2025Social Hatred: Efficient Multimodal Detection of Hatemongerscited by 0
  32. 2025Classification is a RAG problem: A case study on hate speech detectioncited by 0
  33. 2026Aligning Implied Statements for Implicit Hate Speech Generalizability with Context-Bounded Semi-hard Negative Miningcited by 0
  34. 2026A Survey of Toxicity Detection and Mitigation Strategies for Multilingual Language Modelscited by 0
  35. 2026Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languagescited by 0
  36. 2026Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retrainingcited by 0
  37. 2026Introducing the Privacy-HSD Trade-off: Hate Speech Detection, but not at the Cost of Privacycited by 0
  38. 2026SinoGlyphBench: A Diagnostic Benchmark for Chinese Glyph-Level Obfuscation in Language-Model Moderationcited by 0

50 most cited of 51 All 51 (JSON)