mlsanityData to 7 Oct 2026

Investigating Failures of Automatic Translation in the Case of Unambiguous Gender

arXiv 2104.07838 ↗First posted 16 Apr 2021cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Transformer based models are the modern work horses for neural machine translation (NMT), reaching state of the art across several benchmarks. Despite their impressive accuracy, we observe a systemic and rudimentary class of errors made by transformer based models with regards to translating from a language that doesn't mark gender on nouns into others that do. We find that even when the surrounding context provides unambiguous evidence of the appropriate grammatical gender marking, no transformer based model we tested was able to accurately gender occupation nouns systematically. We release an evaluation scheme and dataset for measuring the ability of transformer based NMT models to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences. Our dataset translates from an English source into 20 languages from several different language families. With the availability of this dataset, our hope is that the NMT community can iterate on solutions for this class of especially egregious errors.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 32 of its 64 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

32 of 64 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  3. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  4. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  5. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  7. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  8. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  9. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  10. 2019Evaluating Gender Bias in Machine Translationcited by 247
  11. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  12. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
20 more
  1. 2018Assessing Gender Bias in Machine Translation -- A Case Study with Google Translatecited by 121
  2. 2017Structured Attention Networkscited by 95
  3. 2017Neural Machine Translation with Extended Contextcited by 95
  4. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  5. 2020Multi-Dimensional Gender Bias Classificationcited by 80
  6. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  7. 2019Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translationcited by 45
  8. 2020Mitigating Gender Bias in Machine Translation with Target Gender Annotationscited by 42
  9. 2020Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make Itcited by 40
  10. 2017A Challenge Set Approach to Evaluating Machine Translationcited by 37
  11. 2016Tree-to-Sequence Attentional Neural Machine Translationcited by 28
  12. 2020Automatically Identifying Gender Issues in Machine Translation using Perturbationscited by 28
  13. 2019Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injectioncited by 27
  14. 2017Towards String-to-Tree Neural Machine Translationcited by 19
  15. 2019Unsupervised Discovery of Gendered Language through Latent-Variable Modelingcited by 19
  16. 2019Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translationcited by 15
  17. 2020Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Biascited by 15
  18. 2019Multilingual Graphemic Hybrid ASR with Massive Data Augmentationcited by 13
  19. 2020Astraea: Grammar-based Fairness Testingcited by 11
  20. 2016Word Ordering Without Syntaxcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 4; 2023: 11; 2024: 1.

20 papers.

Sort:

By area:

  1. 2023Large Language Model Alignment: A Surveycited by 241
  2. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  3. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  4. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  5. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  6. 2022Perturbation Augmentation for Fairer NLPcited by 51
  7. 2021Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue modelscited by 23
  8. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  9. 2023Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scalecited by 23
  10. 2023CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Modelscited by 22
  11. 2023Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpuscited by 20
  12. 2023GATE: A Challenge Set for Gender-Ambiguous Translation Examplescited by 18
8 more
  1. 2023Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Modelcited by 14
  2. 2022How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contextscited by 10
  3. 2023Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translationcited by 10
  4. 2023Gender, names and other mysteries: Towards the ambiguous for gender-inclusive translationcited by 7
  5. 2023The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languagescited by 3
  6. 2023Good, but not always Fair: An Evaluation of Gender Bias for three commercial Machine Translation Systemscited by 2
  7. 2022Screening Gender Transfer in Neural Machine Translationcited by 1
  8. 2024Towards Massive Multilingual Holistic Biascited by 0