# Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms

arXiv: https://arxiv.org/abs/2503.04372 · HTML: https://mlsanity.com/paper/2503.04372/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 89 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Builds on (most cited first, 30 of 47)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [UnQovering Stereotyping Biases via Underspecified Questions](https://mlsanity.com/paper/2010.02428/) (2020)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [Debiasing Pre-trained Contextualised Embeddings](https://mlsanity.com/paper/2101.09523/) (2021)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [Getting Gender Right in Neural Machine Translation](https://mlsanity.com/paper/1909.05088/) (2019)
- [Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem](https://mlsanity.com/paper/2004.04498/) (2020)
- [Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias](https://mlsanity.com/paper/2010.14534/) (2020)
- [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020)
- [Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques](https://mlsanity.com/paper/1901.03116/) (2019)
- [On Measuring Gender Bias in Translation of Gender-neutral Pronouns](https://mlsanity.com/paper/1905.11684/) (2019)
- [Gender Bias in Masked Language Models for Multiple Languages](https://mlsanity.com/paper/2205.00551/) (2022)
- [A causal framework for explaining the predictions of black-box sequence-to-sequence models](https://mlsanity.com/paper/1707.01943/) (2017)
- [Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus](https://mlsanity.com/paper/2006.05754/) (2020)

## Built on by (most cited first, 5 of 5)

- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [EuroGEST: Investigating gender stereotypes in multilingual language models](https://mlsanity.com/paper/2506.03867/) (2025)
- [GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metrics](https://mlsanity.com/paper/2510.06841/) (2025)
- [Identity-Robust Language Model Generation via Content Integrity Preservation](https://mlsanity.com/paper/2601.09141/) (2026)
- [Gender Bias in MT for a Genderless Language: New Benchmarks for Basque](https://mlsanity.com/paper/2603.08153/) (2026)

## Abstract (arXiv)

Machine Translation (MT) systems frequently encounter gender-ambiguous occupational terms, where they must assign gender without explicit contextual cues. While individual translations in such cases may not be inherently biased, systematic patterns-such as consistently translating certain professions with specific genders-can emerge, reflecting and perpetuating societal stereotypes. This ambiguity challenges traditional instance-level single-answer evaluation approaches, as no single gold standard translation exists. To address this, we introduce GRAPE, a probability-based metric designed to evaluate gender bias by analyzing aggregated model responses. Alongside this, we present GAMBIT, a benchmarking dataset in English with gender-ambiguous occupational terms. Using GRAPE, we evaluate several MT systems and examine whether their gendered translations in Greek and French align with or diverge from societal stereotypes, real-world occupational gender distributions, and normative standards