# Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition

arXiv: https://arxiv.org/abs/2002.10361 · HTML: https://mlsanity.com/paper/2002.10361/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 30 of 43)

- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [A Survey of Race, Racism, and Anti-Racism in NLP](https://mlsanity.com/paper/2106.11410/) (2021)
- [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022)
- [Towards generalisable hate speech detection: a review on obstacles and solutions](https://mlsanity.com/paper/2102.08886/) (2021)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Handling Bias in Toxic Speech Detection: A Survey](https://mlsanity.com/paper/2202.00126/) (2022)
- [A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challenges](https://mlsanity.com/paper/2305.06969/) (2023)
- [Language-Driven Image Style Transfer](https://mlsanity.com/paper/2106.00178/) (2021)
- [Assessing Multilingual Fairness in Pre-trained Multimodal Representations](https://mlsanity.com/paper/2106.06683/) (2021)
- [Contrastive Learning for Fair Representations](https://mlsanity.com/paper/2109.10645/) (2021)
- [Evaluating Debiasing Techniques for Intersectional Biases](https://mlsanity.com/paper/2109.10441/) (2021)
- [Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold](https://mlsanity.com/paper/2206.09755/) (2022)
- [De-biasing "bias" measurement](https://mlsanity.com/paper/2205.05770/) (2022)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classification](https://mlsanity.com/paper/2105.02778/) (2021)
- [Fairness in Language Models Beyond English: Gaps and Challenges](https://mlsanity.com/paper/2302.12578/) (2023)
- [Adversarial Stylometry in the Wild: Transferable Lexical Substitution Attacks on Author Profiling](https://mlsanity.com/paper/2101.11310/) (2021)
- [MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languages](https://mlsanity.com/paper/2208.13078/) (2022)
- [OxonFair: A Flexible Toolkit for Algorithmic Fairness](https://mlsanity.com/paper/2407.13710/) (2024)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?](https://mlsanity.com/paper/2406.16316/) (2024)
- [APEACH: Attacking Pejorative Expressions with Analysis on Crowd-Generated Hate Speech Evaluation Datasets](https://mlsanity.com/paper/2202.12459/) (2022)
- [Are Pretrained Multilingual Models Equally Fair Across Languages?](https://mlsanity.com/paper/2210.05457/) (2022)
- [Easy Adaptation to Mitigate Gender Bias in Multilingual Text Classification](https://mlsanity.com/paper/2204.05459/) (2022)
- [Highly Generalizable Models for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2201.11294/) (2022)
- ["You Can't Fix What You Can't Measure": Privately Measuring Demographic Performance Disparities in Federated Learning](https://mlsanity.com/paper/2206.12183/) (2022)
- [Human-in-the-Loop Hate Speech Classification in a Multilingual Context](https://mlsanity.com/paper/2212.02108/) (2022)
- [Data Bias According to Bipol: Men are Naturally Right and It is the Role of Women to Follow Their Lead](https://mlsanity.com/paper/2404.04838/) (2024)

## Abstract (arXiv)

Existing research on fairness evaluation of document classification models mainly uses synthetic monolingual data without ground truth for author demographic attributes. In this work, we assemble and publish a multilingual Twitter corpus for the task of hate speech detection with inferred four author demographic factors: age, country, gender and race/ethnicity. The corpus covers five languages: English, Italian, Polish, Portuguese and Spanish. We evaluate the inferred demographic labels with a crowdsourcing platform, Figure Eight. To examine factors that can cause biases, we take an empirical analysis of demographic predictability on the English corpus. We measure the performance of four popular document classifiers and evaluate the fairness and bias of the baseline classifiers on the author-level demographic attributes.