# From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets

arXiv: https://arxiv.org/abs/2404.17874 · HTML: https://mlsanity.com/paper/2404.17874/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 143 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 30 of 52)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [Reducing Gender Bias in Abusive Language Detection](https://mlsanity.com/paper/1808.07231/) (2018)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [A Benchmark Dataset for Learning to Intervene in Online Hate Speech](https://mlsanity.com/paper/1909.04251/) (2019)
- [CONAN -- COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech](https://mlsanity.com/paper/1910.03270/) (2019)
- [Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis](https://mlsanity.com/paper/1701.08118/) (2017)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [Detecting Online Hate Speech Using Context Aware Models](https://mlsanity.com/paper/1710.07395/) (2017)
- [Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning](https://mlsanity.com/paper/2009.10277/) (2020)
- [ETHOS: an Online Hate Speech Detection Dataset](https://mlsanity.com/paper/2006.08328/) (2020)
- [Hate Lingo: A Target-based Linguistic Analysis of Hate Speech in Social Media](https://mlsanity.com/paper/1804.04257/) (2018)
- [Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models](https://mlsanity.com/paper/2206.09917/) (2022)
- [Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hate](https://mlsanity.com/paper/2108.05921/) (2021)
- ["Call me sexist, but...": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples](https://mlsanity.com/paper/2004.12764/) (2020)
- [SemEval-2023 Task 10: Explainable Detection of Online Sexism](https://mlsanity.com/paper/2303.04222/) (2023)
- [Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis](https://mlsanity.com/paper/2010.04543/) (2020)
- [Large-Scale Hate Speech Detection with Cross-Domain Transfer](https://mlsanity.com/paper/2203.01111/) (2022)
- [Detecting East Asian Prejudice on Social Media](https://mlsanity.com/paper/2005.03909/) (2020)

## Built on by (most cited first, 25 of 25)

- [HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter](https://mlsanity.com/paper/2411.15462/) (2024)
- [BanTH: A Multi-label Hate Speech Detection Dataset for Transliterated Bangla](https://mlsanity.com/paper/2410.13281/) (2024)
- [Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?](https://mlsanity.com/paper/2505.18215/) (2025)
- [CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns](https://mlsanity.com/paper/2601.00588/) (2026)
- [Self-Explaining Hate Speech Detection with Moral Rationales](https://mlsanity.com/paper/2601.03481/) (2026)
- [Characterizing Selective Refusal Bias in Large Language Models](https://mlsanity.com/paper/2510.27087/) (2025)
- [When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Edits](https://mlsanity.com/paper/2503.03417/) (2025)
- [BIDWESH: A Bangla Regional Based Hate Speech Detection Dataset](https://mlsanity.com/paper/2507.16183/) (2025)
- [Bridging Gaps in Hate Speech Detection: Meta-Collections and Benchmarks for Low-Resource Iberian Languages](https://mlsanity.com/paper/2510.11167/) (2025)
- [Analyzing Bias in False Refusal Behavior of Large Language Models for Hate Speech Detoxification](https://mlsanity.com/paper/2601.08668/) (2026)
- [HateMirage: An Explainable Multi-Dimensional Dataset for Decoding Faux Hate and Subtle Online Abuse](https://mlsanity.com/paper/2603.02684/) (2026)
- [The Enforcement and Feasibility of Hate Speech Moderation](https://mlsanity.com/paper/2604.12289/) (2026)
- [Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure](https://mlsanity.com/paper/2604.25580/) (2026)
- [HateDebias: On the Diversity and Variability of Hate Speech Debiasing](https://mlsanity.com/paper/2406.04876/) (2024)
- [Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models](https://mlsanity.com/paper/2505.19121/) (2025)
- [Web(er) of Hate: A Survey on How Hate Speech Is Typed](https://mlsanity.com/paper/2506.16190/) (2025)
- [Efficient Hate Speech Detection: A Three-Layer LoRA-Tuned BERTweet Framework](https://mlsanity.com/paper/2511.06051/) (2025)
- [Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task](https://mlsanity.com/paper/2604.14907/) (2026)
- [Explain the Flag: Contextualizing Hate Speech Beyond Censorship](https://mlsanity.com/paper/2604.14970/) (2026)
- [AraDetox: A Multi-Dialect Arabic Detoxification Dataset](https://mlsanity.com/paper/2608.22894/) (2026)
- [AtlasNLP: A Country-Aware Atlas of Dataset Representation in NLP](https://mlsanity.com/paper/2608.30107/) (2026)
- [ParsHate: A Benchmark Dataset for Hate and Target Detection in Persian](https://mlsanity.com/paper/2609.16393/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)
- [GENDIS: Gender-Related Discourse and Stance in Multilingual YouTube Comments from India, Kenya, and Nigeria](https://mlsanity.com/paper/2610.17778/) (2026)
- [From Triage to Impact: A Multi-Hop Reasoning Benchmark for Explainable Persian Hate Speech](https://mlsanity.com/paper/2610.20370/) (2026)

## Abstract (arXiv)

Perceptions of hate can vary greatly across cultural contexts. Hate speech (HS) datasets, however, have traditionally been developed by language. This hides potential cultural biases, as one language may be spoken in different countries home to different cultures. In this work, we evaluate cultural bias in HS datasets by leveraging two interrelated cultural proxies: language and geography. We conduct a systematic survey of HS datasets in eight languages and confirm past findings on their English-language bias, but also show that this bias has been steadily decreasing in the past few years. For three geographically-widespread languages -- English, Arabic and Spanish -- we then leverage geographical metadata from tweets to approximate geo-cultural contexts by pairing language and country information. We find that HS datasets for these languages exhibit a strong geo-cultural bias, largely overrepresenting a handful of countries (e.g., US and UK for English) relative to their prominence in both the broader social media population and the general population speaking these languages. Based on these findings, we formulate recommendations for the creation of future HS datasets.