mlsanityData to 14 Oct 2026

What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns

arXiv 2305.16051 ↗First posted 25 May 2023cs.CL cs.AI cs.CYcited by 19

Not on the map. Why?

Abstract (arXiv)

As 3rd-person pronoun usage shifts to include novel forms, e.g., neopronouns, we need more research on identity-inclusive NLP. Exclusion is particularly harmful in one of the most popular NLP applications, machine translation (MT). Wrong pronoun translations can discriminate against marginalized groups, e.g., non-binary individuals (Dev et al., 2021). In this ``reality check'', we study how three commercial MT systems translate 3rd-person pronouns. Concretely, we compare the translations of gendered vs. gender-neutral pronouns from English to five other languages (Danish, Farsi, French, German, Italian), and vice versa, from Danish to English. Our error analysis shows that the presence of a gender-neutral pronoun often leads to grammatical and semantic translation errors. Similarly, gender neutrality is often not preserved. By surveying the opinions of affected native speakers from diverse languages, we provide recommendations to address the issue in future MT research.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

30 of 45 references.

References · 30

  1. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  2. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  3. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  4. 2018Gender Bias in Coreference Resolutioncited by 442
  5. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  6. 2019Evaluating Gender Bias in Machine Translationcited by 249
  7. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  8. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  9. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  10. 2021Gender Bias in Machine Translationcited by 117
  11. 2020Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problemcited by 86
  12. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
18 more
  1. 2019Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniquescited by 65
  2. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  3. 2019On Measuring Gender Bias in Translation of Gender-neutral Pronounscited by 60
  4. 2022Perturbation Augmentation for Fairer NLPcited by 52
  5. 2022Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gendercited by 47
  6. 2020Mitigating Gender Bias in Machine Translation with Target Gender Annotationscited by 42
  7. 2020Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make Itcited by 40
  8. 2019A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spacescited by 36
  9. 2022Choose Your Lenses: Flaws in Gender Bias Evaluationcited by 29
  10. 2020Automatically Identifying Gender Issues in Machine Translation using Perturbationscited by 28
  11. 2021Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translationcited by 28
  12. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 23
  13. 2021Evaluating Gender Bias in Hindi-English Machine Translationcited by 19
  14. 2022Fair and Argumentative Language Modeling for Computational Argumentationcited by 18
  15. 2021GENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomenacited by 16
  16. 2022How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronounscited by 16
  17. 2021Examining Covert Gender Bias: A Case Study in Turkish and English Machine Translation Modelscited by 15
  18. 2021Extending Challenge Sets to Uncover Gender Bias in Machine Translation: Impact of Stereotypical Verbs and Adjectivescited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 3; 2026: 1.

19 papers.

Sort:

By area:

  1. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  2. 2023Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpuscited by 20
  3. 2023A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translationcited by 19
  4. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  5. 2024Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help Youcited by 15
  6. 2024What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Studycited by 15
  7. 2024A Prompt Response to the Demand for Automatic Gender-Neutral Translationcited by 12
  8. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  9. 2024Enhancing Gender-Inclusive Machine Translation with Neomorphemes and Large Language Modelscited by 7
  10. 2023Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INEScited by 6
  11. 2024MiTTenS: A Dataset for Evaluating Gender Mistranslationcited by 5
  12. 2024Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMscited by 5
7 more
  1. 2024The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classificationcited by 5
  2. 2025Gender-Neutral Machine Translation Strategies in Practicecited by 3
  3. 2024Transforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronounscited by 2
  4. 2025Uncertainty Quantification for Evaluating Machine Translation Biascited by 2
  5. 2024WMT24 Test Suite: Gender Resolution in Speaker-Listener Dialogue Rolescited by 1
  6. 2025Tell Me What You Know About Sexism: Expert-LLM Interaction Strategies and Co-Created Definitions for Zero-Shot Sexism Detectioncited by 0
  7. 2026GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in Germancited by 0