mlsanityData to 7 Oct 2026

Bias Beyond English: Counterfactual Tests for Bias in Sentiment Analysis in Four Languages

arXiv 2305.11673 ↗First posted 19 May 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Sentiment analysis (SA) systems are used in many products and hundreds of languages. Gender and racial biases are well-studied in English SA systems, but understudied in other languages, with few resources for such studies. To remedy this, we build a counterfactual evaluation corpus for gender and racial/migrant bias in four languages. We demonstrate its usefulness by answering a simple but important question that an engineer might need to answer when deploying a system: What biases do systems import from pre-trained models when compared to a baseline with no pre-training? Our evaluation corpus, by virtue of being counterfactual, not only reveals which models have less bias, but also pinpoints changes in model bias behaviour, which enables more targeted mitigation strategies. We release our code and evaluation corpora to facilitate future research.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 12 of its 35 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

12 of 35 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  4. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  5. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  6. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  7. 2020The Multilingual Amazon Reviews Corpuscited by 142
  8. 2020Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Researchcited by 67
  9. 2022Quantifying Social Biases Using Templates is Unreliablecited by 38
  10. 2020Unsupervised Bitext Mining and Translation via Self-trained Contextual Embeddingscited by 16
  11. 2021Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluationcited by 9
  12. 2022Measuring Fairness of Text Classifiers via Prediction Sensitivitycited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 1.

8 papers.

  1. 2024Cognitive Bias in Decision-Making with LLMscited by 104
  2. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  3. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 32
  4. 2024The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Modelscited by 6
  5. 2023Cross-lingual Transfer Can Worsen Bias in Sentiment Analysiscited by 3
  6. 2024Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Modelscited by 1
  7. 2024MAFIA: Multi-Adapter Fused Inclusive LanguAge Modelscited by 0
  8. 2025Human-Centered Supervision for Sentiment Analysis in Telugu: A Systematic Inquiry Beyond Accuracycited by 0