# HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter

arXiv: https://arxiv.org/abs/2411.15462 · HTML: https://mlsanity.com/paper/2411.15462/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 93 references that are in the map.
- First posted on arXiv 23 Nov 2024.

## Builds on (most cited first, 30 of 41)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](https://mlsanity.com/paper/2202.11176/) (2022)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models](https://mlsanity.com/paper/2206.09917/) (2022)
- [Deep Learning Models for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2004.06465/) (2020)
- [On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research](https://mlsanity.com/paper/2304.12397/) (2023)
- [Handling and Presenting Harmful Text in NLP Research](https://mlsanity.com/paper/2204.14256/) (2022)
- [Large-Scale Hate Speech Detection with Cross-Domain Transfer](https://mlsanity.com/paper/2203.01111/) (2022)
- [Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation](https://mlsanity.com/paper/2402.12649/) (2024)
- [Hostility Detection Dataset in Hindi](https://mlsanity.com/paper/2011.03588/) (2020)
- [Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media](https://mlsanity.com/paper/2104.06999/) (2021)
- [Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Media](https://mlsanity.com/paper/2305.16941/) (2023)
- [From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets](https://mlsanity.com/paper/2404.17874/) (2024)
- [Data Bootstrapping Approaches to Improve Low Resource Abusive Language Detection for Indic Languages](https://mlsanity.com/paper/2204.12543/) (2022)
- [Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments](https://mlsanity.com/paper/2203.11764/) (2022)
- [Toxic Bias: Perspective API Misreads German as More Toxic](https://mlsanity.com/paper/2312.12651/) (2023)
- [On Cross-Dataset Generalization in Automatic Detection of Online Abuse](https://mlsanity.com/paper/2010.07414/) (2020)
- [NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps](https://mlsanity.com/paper/2404.01651/) (2024)
- [Just Another Day on Twitter: A Complete 24 Hours of Twitter Data](https://mlsanity.com/paper/2301.11429/) (2023)
- [Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset](https://mlsanity.com/paper/2403.19559/) (2024)
- [HERDPhobia: A Dataset for Hate Speech against Fulani in Nigeria](https://mlsanity.com/paper/2211.15262/) (2022)

## Built on by (most cited first, 13 of 13)

- [Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs](https://mlsanity.com/paper/2601.03087/) (2026)
- [Community Moderation and the New Epistemology of Fact Checking on Social Media](https://mlsanity.com/paper/2505.20067/) (2025)
- [ToxiFrench: Benchmarking and Enhancing Language Models via CoT Fine-Tuning for French Toxicity Detection](https://mlsanity.com/paper/2508.11281/) (2025)
- [Compositional Generalisation for Explainable Hate Speech Detection](https://mlsanity.com/paper/2506.03916/) (2025)
- [The Enforcement and Feasibility of Hate Speech Moderation](https://mlsanity.com/paper/2604.12289/) (2026)
- [Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure](https://mlsanity.com/paper/2604.25580/) (2026)
- [Manipulation-Proof Oblivious Audits against Deceptive Model Providers](https://mlsanity.com/paper/2608.04365/) (2026)
- [HatePrototypes: Interpretable and Transferable Representations for Implicit and Explicit Hate Speech Detection](https://mlsanity.com/paper/2511.06391/) (2025)
- [Boosting Accuracy and Interpretability in Multilingual Hate Speech Detection Through Layer Freezing and Explainable AI](https://mlsanity.com/paper/2601.02697/) (2026)
- [ArabDiscrim: A Decade-Long Arabic Facebook Corpus on Racism and Discrimination](https://mlsanity.com/paper/2605.22081/) (2026)
- [SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue](https://mlsanity.com/paper/2609.01548/) (2026)
- [ParsHate: A Benchmark Dataset for Hate and Target Detection in Persian](https://mlsanity.com/paper/2609.16393/) (2026)
- [GENDIS: Gender-Related Discourse and Stance in Multilingual YouTube Comments from India, Kenya, and Nigeria](https://mlsanity.com/paper/2610.17778/) (2026)

## Abstract (arXiv)

To address the global challenge of online hate speech, prior research has developed detection models to flag such content on social media. However, due to systematic biases in evaluation datasets, the real-world effectiveness of these models remains unclear, particularly across geographies. We introduce HateDay, the first global hate speech dataset representative of social media settings, constructed from a random sample of all tweets posted on September 21, 2022 and covering eight languages and four English-speaking countries. Using HateDay, we uncover substantial variation in the prevalence and composition of hate speech across languages and regions. We show that evaluations on academic datasets greatly overestimate real-world detection performance, which we find is very low, especially for non-European languages. Our analysis identifies key drivers of this gap, including models' difficulty to distinguish hate from offensive speech and a mismatch between the target groups emphasized in academic datasets and those most frequently targeted in real-world settings. We argue that poor model performance makes public models ill-suited for automatic hate speech moderation and find that high moderation rates are only achievable with substantial human oversight. Our results underscore the need to evaluate detection systems on data that reflects the complexity and diversity of real-world social media.