mlsanityData to 14 Oct 2026

Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition

arXiv 2002.10361 ↗First posted 24 Feb 2020cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Existing research on fairness evaluation of document classification models mainly uses synthetic monolingual data without ground truth for author demographic attributes. In this work, we assemble and publish a multilingual Twitter corpus for the task of hate speech detection with inferred four author demographic factors: age, country, gender and race/ethnicity. The corpus covers five languages: English, Italian, Polish, Portuguese and Spanish. We evaluate the inferred demographic labels with a crowdsourcing platform, Figure Eight. To examine factors that can cause biases, we take an empirical analysis of demographic predictability on the English corpus. We measure the performance of four popular document classifiers and evaluate the fairness and bias of the baseline classifiers on the author-level demographic attributes.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 13; 2023: 4; 2024: 8; 2025: 4; 2026: 3.

43 papers.

Sort:

By area:

  1. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  2. 2021Trustworthy AI: A Computational Perspectivecited by 93
  3. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  4. 2021A Survey of Race, Racism, and Anti-Racism in NLPcited by 75
  5. 2022Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotationcited by 69
  6. 2021Towards generalisable hate speech detection: a review on obstacles and solutionscited by 66
  7. 2022Theories of "Gender" in NLP Bias Researchcited by 62
  8. 2022Handling Bias in Toxic Speech Detection: A Surveycited by 61
  9. 2023A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challengescited by 41
  10. 2021Language-Driven Image Style Transfercited by 37
  11. 2021Assessing Multilingual Fairness in Pre-trained Multimodal Representationscited by 35
  12. 2021Contrastive Learning for Fair Representationscited by 34
31 more
  1. 2021Evaluating Debiasing Techniques for Intersectional Biasescited by 30
  2. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  3. 2022De-biasing "bias" measurementcited by 20
  4. 2024Fairness and Bias in Multimodal AI: A Surveycited by 20
  5. 2021The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classificationcited by 18
  6. 2023Fairness in Language Models Beyond English: Gaps and Challengescited by 16
  7. 2021Adversarial Stylometry in the Wild: Transferable Lexical Substitution Attacks on Author Profilingcited by 13
  8. 2022MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languagescited by 13
  9. 2024OxonFair: A Flexible Toolkit for Algorithmic Fairnesscited by 12
  10. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  11. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 10
  12. 2022APEACH: Attacking Pejorative Expressions with Analysis on Crowd-Generated Hate Speech Evaluation Datasetscited by 8
  13. 2022Are Pretrained Multilingual Models Equally Fair Across Languages?cited by 8
  14. 2022Easy Adaptation to Mitigate Gender Bias in Multilingual Text Classificationcited by 5
  15. 2022Highly Generalizable Models for Multilingual Hate Speech Detectioncited by 3
  16. 2022"You Can't Fix What You Can't Measure": Privately Measuring Demographic Performance Disparities in Federated Learningcited by 3
  17. 2022Human-in-the-Loop Hate Speech Classification in a Multilingual Contextcited by 3
  18. 2024Data Bias According to Bipol: Men are Naturally Right and It is the Role of Women to Follow Their Leadcited by 2
  19. 2024Synthetic Data Generation for Intersectional Fairness by Leveraging Hierarchical Group Structurecited by 2
  20. 2025From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Textcited by 2
  21. 2026Spotlights and Blindspots: Evaluating Machine-Generated Text Detectioncited by 2
  22. 2023Fair Without Leveling Down: A New Intersectional Fairness Definitioncited by 1
  23. 2024English offensive text detection using CNN based Bi-GRU modelcited by 1
  24. 2025Evolving Hate Speech Online: An Adaptive Framework for Detection and Mitigationcited by 1
  25. 2025Iterative Multilingual Spectral Attribute Erasurecited by 1
  26. 2026Cross-Language Speaker Attribute Prediction Using MIL and RLcited by 1
  27. 2026Multilingual Hate Speech Detection and Counterspeech Generation: A Comprehensive Survey and Practical Guidecited by 1
  28. 2022Suum Cuique: Studying Bias in Taboo Detection with a Community Perspectivecited by 0
  29. 2023Model and Evaluation: Towards Fairness in Multilingual Text Classificationcited by 0
  30. 2024HateDebias: On the Diversity and Variability of Hate Speech Debiasingcited by 0
  31. 2025Demographic Biases and Gaps in the Perception of Sexism in Large Language Modelscited by 0