Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms

arXiv 2503.04372 ↗First posted 6 Mar 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Machine Translation (MT) systems frequently encounter gender-ambiguous occupational terms, where they must assign gender without explicit contextual cues. While individual translations in such cases may not be inherently biased, systematic patterns-such as consistently translating certain professions with specific genders-can emerge, reflecting and perpetuating societal stereotypes. This ambiguity challenges traditional instance-level single-answer evaluation approaches, as no single gold standard translation exists. To address this, we introduce GRAPE, a probability-based metric designed to evaluate gender bias by analyzing aggregated model responses. Alongside this, we present GAMBIT, a benchmarking dataset in English with gender-ambiguous occupational terms. Using GRAPE, we evaluate several MT systems and examine whether their gendered translations in Greek and French align with or diverge from societal stereotypes, real-world occupational gender distributions, and normative standards

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 47 of its 89 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

47 of 89 references.

References · 47

2024A Survey on LLM-as-a-Judgecited by 1,384
2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
2020COMET: A Neural Framework for MT Evaluationcited by 1,024
2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
2020Beyond English-Centric Multilingual Machine Translationcited by 559
2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
2018Gender Bias in Coreference Resolutioncited by 438

35 more

2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
2019Evaluating Gender Bias in Machine Translationcited by 247
2018Gender Bias in Neural Natural Language Processingcited by 210
2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
2018Assessing Gender Bias in Machine Translation -- A Case Study with Google Translatecited by 121
2021Gender Bias in Machine Translationcited by 116
2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
2021Debiasing Pre-trained Contextualised Embeddingscited by 107
2024EuroLLM: Multilingual Language Models for Europecited by 103
2019Getting Gender Right in Neural Machine Translationcited by 98
2020Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problemcited by 86
2020Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Biascited by 82
2020Multi-Dimensional Gender Bias Classificationcited by 80
2019Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniquescited by 65
2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
2019On Measuring Gender Bias in Translation of Gender-neutral Pronounscited by 59
2017A causal framework for explaining the predictions of black-box sequence-to-sequence modelscited by 58
2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
2023MISGENDERED: Limits of Large Language Models in Understanding Pronounscited by 43
2019Examining Gender Bias in Languages with Grammatical Gendercited by 40
2022MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translationcited by 38
2021Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgettingcited by 34
2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33
2020Automatically Identifying Gender Issues in Machine Translation using Perturbationscited by 28
2023Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challengescited by 22
2022Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruningcited by 20
2023Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpuscited by 20
2021GENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomenacited by 16
2022Don't Forget About Pronouns: Removing Gender Bias in Language Models Without Losing Factual Gender Informationcited by 16
2024What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Studycited by 15
2023Measuring Gender Bias in West Slavic Language Modelscited by 10
2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
2020Mitigating Gender Bias in Machine Learning Data Setscited by 5
2024GOSt-MT: A Knowledge Graph for Occupation-related Gender Biases in Machine Translationcited by 4
2024Generating Gender Alternatives in Machine Translationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

2025Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translationcited by 5
2025EuroGEST: Investigating gender stereotypes in multilingual language modelscited by 4
2025GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metricscited by 4
2026Identity-Robust Language Model Generation via Content Integrity Preservationcited by 1
2026Gender Bias in MT for a Genderless Language: New Benchmarks for Basquecited by 0