mlsanityData to 7 Oct 2026

Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation

arXiv 2410.10995 ↗First posted 14 Oct 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Quality estimation (QE)-the automatic assessment of translation quality-has recently become crucial across several stages of the translation pipeline, from data curation to training and decoding. While QE metrics have been optimized to align with human judgments, whether they encode social biases has been largely overlooked. Biased QE risks favoring certain demographic groups over others, e.g., by exacerbating gaps in visibility and usability. This paper defines and investigates gender bias of QE metrics and discusses its downstream implications for machine translation (MT). Experiments with state-of-the-art QE metrics across multiple domains, datasets, and languages reveal significant bias. When a human entity's gender in the source is undisclosed, masculine-inflected translations score higher than feminine-inflected ones, and gender-neutral translations are penalized. Even when contextual cues disambiguate gender, using context-aware QE metrics leads to more errors in selecting the correct translation inflection for feminine referents than for masculine ones. Moreover, a biased QE metric affects data filtering and quality-aware decoding. Our findings underscore the need for a renewed focus on developing and evaluating QE metrics centered on gender.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 63 of its 100 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

63 of 100 references.

References · 63

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  9. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  10. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  11. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  12. 2019On Measuring Social Biases in Sentence Encoderscited by 397
38 more
  1. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  2. 2019Evaluating Gender Bias in Machine Translationcited by 247
  3. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  4. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  5. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  6. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  7. 2021Gender Bias in Machine Translationcited by 116
  8. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  9. 2019Getting Gender Right in Neural Machine Translationcited by 98
  10. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  11. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  12. 2020Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problemcited by 86
  13. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  14. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  15. 2019Putting Fairness Principles into Practice: Challenges, Metrics, and Improvementscited by 70
  16. 2019Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniquescited by 65
  17. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  18. 2019On Measuring Gender Bias in Translation of Gender-neutral Pronounscited by 59
  19. 2021BlonDe: An Automatic Evaluation Metric for Document-level Machine Translationcited by 58
  20. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  21. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  22. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  23. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  24. 2020Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make Itcited by 40
  25. 2022MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translationcited by 38
  26. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Modelcited by 33
  27. 2023Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventionscited by 28
  28. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  29. 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
  30. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  31. 2024Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gapscited by 24
  32. 2019Towards Understanding Gender Bias in Relation Extractioncited by 23
  33. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  34. 2023Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpuscited by 20
  35. 2023What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronounscited by 19
  36. 2023A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translationcited by 19
  37. 2023There's no Data Like Better Data: Using QE Metrics for MT Data Filteringcited by 19
  38. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inferencecited by 19

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 5.

10 papers.

  1. 2025Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Termscited by 5
  2. 2025GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metricscited by 4
  3. 2025An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluationcited by 2
  4. 2025Man Made Language Models? Evaluating LLMs' Perpetuation of Masculine Generics Biascited by 1
  5. 2026FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimationcited by 1
  6. 2026Is a Document Educational or Just Wikipedia-Style? -- Pitfalls of Classifier-Based Quality Filteringcited by 1
  7. 2024MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluationcited by 0
  8. 2026Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarioscited by 0
  9. 2026Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimationcited by 0
  10. 2026Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupationscited by 0