mlsanityData to 7 Oct 2026

Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INES

arXiv 2310.19345 ↗First posted 30 Oct 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

As part of the WMT-2023 "Test suites" shared task, in this paper we summarize the results of two test suites evaluations: MuST-SHE-WMT23 and INES. By focusing on the en-de and de-en language pairs, we rely on these newly created test suites to investigate systems' ability to translate feminine and masculine gender and produce gender-inclusive translations. Furthermore we discuss metrics associated with our test suites and validate them by means of human evaluations. Our results indicate that systems achieve reasonable and comparable performance in correctly translating both feminine and masculine gender forms for naturalistic gender phenomena. Instead, the generation of inclusive language forms in translation emerges as a challenging task for all the evaluated MT models, indicating room for future improvements and research on the topic.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

23 of 35 references.

References · 23

  1. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  2. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  3. 2019Evaluating Gender Bias in Machine Translationcited by 247
  4. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  5. 2021Gender Bias in Machine Translationcited by 116
  6. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  7. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  8. 2019On Measuring Gender Bias in Translation of Gender-neutral Pronounscited by 59
  9. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  10. 2023"I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generationcited by 56
  11. 2022Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gendercited by 47
  12. 2021NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternativescited by 46
11 more
  1. 2020Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make Itcited by 40
  2. 2021First the worst: Finding better gender translations during beam searchcited by 28
  3. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  4. 2023Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challengescited by 22
  5. 2023What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronounscited by 19
  6. 2023GATE: A Challenge Set for Gender-Ambiguous Translation Examplescited by 18
  7. 2020Breeding Gender-aware Direct Speech Translation Systemscited by 16
  8. 2022How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronounscited by 16
  9. 2023Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Modelcited by 14
  10. 2021How to Split: the Effect of Word Segmentation on Gender Bias in Speech Translationcited by 11
  11. 2021Improving Gender Translation Accuracy with Filtered Self-Trainingcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 1.

6 papers.

  1. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  2. 2024What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Studycited by 15
  3. 2024A Prompt Response to the Demand for Automatic Gender-Neutral Translationcited by 12
  4. 2024Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMscited by 5
  5. 2025Gender-Neutral Machine Translation Strategies in Practicecited by 3
  6. 2024WMT24 Test Suite: Gender Resolution in Speaker-Listener Dialogue Rolescited by 1