# Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate

arXiv: https://arxiv.org/abs/1809.02208 · HTML: https://mlsanity.com/paper/1809.02208/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2018.

## Built on by (most cited first, 30 of 121)

- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models](https://mlsanity.com/paper/2304.03738/) (2023)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation](https://mlsanity.com/paper/2109.03858/) (2021)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Perturbation Augmentation for Fairer NLP](https://mlsanity.com/paper/2205.12586/) (2022)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [A Causal Lens for Controllable Text Generation](https://mlsanity.com/paper/2201.09119/) (2022)
- [Lost in Translation: Large Language Models in Non-English Content Analysis](https://mlsanity.com/paper/2306.07377/) (2023)
- [ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages](https://mlsanity.com/paper/2305.10510/) (2023)
- [Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice](https://mlsanity.com/paper/2210.03709/) (2022)
- [Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophy](https://mlsanity.com/paper/2205.12771/) (2022)
- [Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans](https://mlsanity.com/paper/2401.08495/) (2024)
- [Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models](https://mlsanity.com/paper/2109.03300/) (2021)
- [Robustness Analysis of Video-Language Models Against Visual and Language Perturbations](https://mlsanity.com/paper/2207.02159/) (2022)
- [Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models](https://mlsanity.com/paper/2210.06475/) (2022)
- [Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challenges](https://mlsanity.com/paper/2301.10075/) (2023)
- [Investigating Failures of Automatic Translation in the Case of Unambiguous Gender](https://mlsanity.com/paper/2104.07838/) (2021)
- [Dataset for Identification of Homophobia and Transophobia in Multilingual YouTube Comments](https://mlsanity.com/paper/2109.00227/) (2021)
- [A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation](https://mlsanity.com/paper/2310.12127/) (2023)
- [The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classification](https://mlsanity.com/paper/2105.02778/) (2021)
- [Ethics Sheets for AI Tasks](https://mlsanity.com/paper/2107.01183/) (2021)
- [Refocusing on Relevance: Personalization in NLG](https://mlsanity.com/paper/2109.05140/) (2021)
- [GATE: A Challenge Set for Gender-Ambiguous Translation Examples](https://mlsanity.com/paper/2303.03975/) (2023)

## Abstract (arXiv)

Recently there has been a growing concern about machine bias, where trained statistical models grow to reflect controversial societal asymmetries, such as gender or racial bias. A significant number of AI tools have recently been suggested to be harmfully biased towards some minority, with reports of racist criminal behavior predictors, Iphone X failing to differentiate between two Asian people and Google photos' mistakenly classifying black people as gorillas. Although a systematic study of such biases can be difficult, we believe that automated translation tools can be exploited through gender neutral languages to yield a window into the phenomenon of gender bias in AI. In this paper, we start with a comprehensive list of job positions from the U.S. Bureau of Labor Statistics (BLS) and used it to build sentences in constructions like "He/She is an Engineer" in 12 different gender neutral languages such as Hungarian, Chinese, Yoruba, and several others. We translate these sentences into English using the Google Translate API, and collect statistics about the frequency of female, male and gender-neutral pronouns in the translated output. We show that GT exhibits a strong tendency towards male defaults, in particular for fields linked to unbalanced gender distribution such as STEM jobs. We ran these statistics against BLS' data for the frequency of female participation in each job position, showing that GT fails to reproduce a real-world distribution of female workers. We provide experimental evidence that even if one does not expect in principle a 50:50 pronominal gender distribution, GT yields male defaults much more frequently than what would be expected from demographic data alone. We are hopeful that this work will ignite a debate about the need to augment current statistical translation tools with debiasing techniques which can already be found in the scientific literature.