mlsanityData to 7 Oct 2026

MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation

arXiv 2211.01355 ↗First posted 2 Nov 2022cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

As generic machine translation (MT) quality has improved, the need for targeted benchmarks that explore fine-grained aspects of quality has increased. In particular, gender accuracy in translation can have implications in terms of output fluency, translation accuracy, and ethics. In this paper, we introduce MT-GenEval, a benchmark for evaluating gender accuracy in translation from English into eight widely-spoken languages. MT-GenEval complements existing benchmarks by providing realistic, gender-balanced, counterfactual data in eight language pairs where the gender of individuals is unambiguous in the input segment, including multi-sentence segments requiring inter-sentential gender agreement. Our data and code is publicly available under a CC BY SA 3.0 license.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

29 of 46 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  5. 2018Gender Bias in Coreference Resolutioncited by 438
  6. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  7. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  8. 2019Evaluating Gender Bias in Machine Translationcited by 247
  9. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  10. 2021Gender Bias in Machine Translationcited by 116
  11. 2019Getting Gender Right in Neural Machine Translationcited by 98
  12. 2017Neural Machine Translation with Extended Contextcited by 95
17 more
  1. 2020Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problemcited by 86
  2. 2019Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniquescited by 65
  3. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  4. 2016Personalized Machine Translation: Preserving Original Author Traitscited by 47
  5. 2019When and Why is Document-level Context Useful in Neural Machine Translation?cited by 45
  6. 2020Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make Itcited by 40
  7. 2017A Challenge Set Approach to Evaluating Machine Translationcited by 37
  8. 2020Evaluating Robustness to Input Perturbations for Neural Machine Translationcited by 36
  9. 2021Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translationcited by 28
  10. 2022CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formalitycited by 24
  11. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  12. 2020Gender Coreference and Bias Evaluation at WMT 2020cited by 21
  13. 2021GENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomenacited by 16
  14. 2021Extending Challenge Sets to Uncover Gender Bias in Machine Translation: Impact of Stereotypical Verbs and Adjectivescited by 14
  15. 2019GeBioToolkit: Automatic Extraction of Gender-Balanced Multilingual Corpus of Wikipedia Biographiescited by 11
  16. 2022A baseline revisited: Pushing the limits of multi-segment models for context-aware translationcited by 11
  17. 2022Sockeye 3: Fast Neural Machine Translation with PyTorchcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 13; 2025: 6; 2026: 8.

38 papers.

Sort:

By area:

  1. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  2. 2023Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpuscited by 20
  3. 2023GATE: A Challenge Set for Gender-Ambiguous Translation Examplescited by 18
  4. 2024What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Studycited by 15
  5. 2023Identifying Context-Dependent Translations for Evaluation Set Productioncited by 14
  6. 2023Document-Level Language Models for Machine Translationcited by 12
  7. 2023Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translationcited by 10
  8. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  9. 2023RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translationcited by 9
  10. 2023Improving Long Context Document-Level Machine Translationcited by 8
  11. 2024Enhancing Gender-Inclusive Machine Translation with Neomorphemes and Large Language Modelscited by 7
  12. 2024Context-Aware or Context-Insensitive? Assessing LLMs' Performance in Document-Level Translationcited by 5
26 more
  1. 2025Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Termscited by 5
  2. 2023Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translationcited by 4
  3. 2024GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languagescited by 4
  4. 2025FairTranslate: An English-French Dataset for Gender Bias Evaluation in Machine Translation by Overcoming Gender Binaritycited by 4
  5. 2025GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metricscited by 4
  6. 2024Generating Gender Alternatives in Machine Translationcited by 3
  7. 2025Gender Bias in English-to-Greek Machine Translationcited by 3
  8. 2023Good, but not always Fair: An Evaluation of Gender Bias for three commercial Machine Translation Systemscited by 2
  9. 2025Uncertainty Quantification for Evaluating Machine Translation Biascited by 2
  10. 2024Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Wordscited by 1
  11. 2026FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimationcited by 1
  12. 2026Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intentcited by 1
  13. 2023On Search Strategies for Document-Level Neural Machine Translationcited by 0
  14. 2023Don't Overlook the Grammatical Gender: Bias Evaluation for Hindi-English Machine Translationcited by 0
  15. 2024Promoting Target Data in Context-aware Neural Machine Translationcited by 0
  16. 2024Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferencescited by 0
  17. 2024Does Context Help Mitigate Gender Bias in Neural Machine Translation?cited by 0
  18. 2024Fine-grained Gender Control in Machine Translation with Large Language Modelscited by 0
  19. 2024MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluationcited by 0
  20. 2025Probing Gender Bias in Multilingual LLMs: A Case Study of Stereotypes in Persiancited by 0
  21. 2026MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generationcited by 0
  22. 2026Cultural Fidelity in English-to-Hindi Translation: A Preservation-Fluency Frontier for Gender Recoverabilitycited by 0
  23. 2026Explaining GAND: A Resource on Gender-Ambiguous Natural Data & Contrastive Attributioncited by 0
  24. 2026Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupationscited by 0
  25. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0
  26. 2026Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Followingcited by 0