mlsanityData to 7 Oct 2026

Evaluating Gender Bias in Machine Translation

arXiv 1906.00591 ↗First posted 3 Jun 2019cs.CLcited by 247

Not on the map. Why?

Abstract (arXiv)

We present the first challenge set and evaluation protocol for the analysis of gender bias in machine translation (MT). Our approach uses two recent coreference resolution datasets composed of English sentences which cast participants into non-stereotypical gender roles (e.g., "The doctor asked the nurse to help her in the operation"). We devise an automatic gender bias evaluation method for eight target languages with grammatical gender, based on morphological analysis (e.g., the use of female inflection for the word "doctor"). Our analyses show that four popular industrial MT systems and two recent state-of-the-art academic MT models are significantly prone to gender-biased translation errors for all tested target languages. Our data and code are made publicly available.

Key facts. Cited by 247 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 43; 2023: 55; 2024: 62; 2025: 30; 2026: 16.

247 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  3. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  4. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  5. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  8. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  9. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  10. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  11. 2021Gender Bias in Machine Translationcited by 116
  12. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
38 more
  1. 2021Trustworthy AI: A Computational Perspectivecited by 93
  2. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  3. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  4. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  5. 2021On Measures of Biases and Harms in NLPcited by 81
  6. 2021FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoderscited by 75
  7. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
  8. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  9. 2021Can We Automate Scientific Reviewing?cited by 63
  10. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  11. 2022Theories of "Gender" in NLP Bias Researchcited by 62
  12. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  13. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  14. 2022A Causal Lens for Controllable Text Generationcited by 45
  15. 2023MISGENDERED: Limits of Large Language Models in Understanding Pronounscited by 43
  16. 2024Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attributioncited by 39
  17. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 39
  18. 2022MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translationcited by 38
  19. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 38
  20. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  21. 2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33
  22. 2023ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languagescited by 33
  23. 2024Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Modelscited by 33
  24. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  25. 2022Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophycited by 31
  26. 2024Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humanscited by 31
  27. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  28. 2023Social Bias Probing: Fairness Benchmarking for Language Modelscited by 29
  29. 2021First the worst: Finding better gender translations during beam searchcited by 28
  30. 2021Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translationcited by 28
  31. 2021Why don't people use character-level machine translation?cited by 27
  32. 2022Task Ambiguity in Humans and Language Modelscited by 27
  33. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  34. 2024Identifying Implicit Social Biases in Vision-Language Modelscited by 27
  35. 2023Ethical Considerations for Machine Translation of Indigenous Languages: Giving a Voice to the Speakerscited by 26
  36. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  37. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  38. 2022The Defeat of the Winograd Schema Challengecited by 25

50 most cited of 247 All 247 (JSON)