# ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages

arXiv: https://arxiv.org/abs/2305.10510 · HTML: https://mlsanity.com/paper/2305.10510/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 247 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 30 of 35)

- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale](https://mlsanity.com/paper/2211.03759/) (2022)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases](https://mlsanity.com/paper/2006.03955/) (2020)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Empowering News Recommendation with Pre-trained Language Models](https://mlsanity.com/paper/2104.07413/) (2021)
- [Many-to-English Machine Translation Tools, Data, and Pretrained Models](https://mlsanity.com/paper/2104.00290/) (2021)
- [Language Model is All You Need: Natural Language Understanding as Question Answering](https://mlsanity.com/paper/2011.03023/) (2020)
- [Markedness in Visual Semantic AI](https://mlsanity.com/paper/2205.11378/) (2022)
- [Self-Supervised Euphemism Detection and Identification for Content Moderation](https://mlsanity.com/paper/2103.16808/) (2021)
- [Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics](https://mlsanity.com/paper/2206.03390/) (2022)
- [Gender Aware Spoken Language Translation Applied to English-Arabic](https://mlsanity.com/paper/1802.09287/) (2018)
- [Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News Detection](https://mlsanity.com/paper/2101.05509/) (2021)
- [Examining Covert Gender Bias: A Case Study in Turkish and English Machine Translation Models](https://mlsanity.com/paper/2108.10379/) (2021)
- [What Do We Mean by "Accessibility Research"? A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019](https://mlsanity.com/paper/2101.04271/) (2021)
- [Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection](https://mlsanity.com/paper/2111.07819/) (2021)

## Built on by (most cited first, 30 of 34)

- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice](https://mlsanity.com/paper/2402.01864/) (2024)
- [GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing](https://mlsanity.com/paper/2407.00600/) (2024)
- [Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implications](https://mlsanity.com/paper/2307.09162/) (2023)
- [Evaluating Biased Attitude Associations of Language Models in an Intersectional Context](https://mlsanity.com/paper/2307.03360/) (2023)
- [Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus](https://mlsanity.com/paper/2310.05294/) (2023)
- [TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties](https://mlsanity.com/paper/2308.03051/) (2023)
- [From Melting Pots to Misrepresentations: Exploring Harms in Generative AI](https://mlsanity.com/paper/2403.10776/) (2024)
- [A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Bias](https://mlsanity.com/paper/2505.09056/) (2025)
- [The Multilingual Divide and Its Impact on Global AI Safety](https://mlsanity.com/paper/2505.21344/) (2025)
- [Evaluating and Mitigating Linguistic Discrimination in Large Language Models](https://mlsanity.com/paper/2404.18534/) (2024)
- ['Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMs](https://mlsanity.com/paper/2409.13484/) (2024)
- [In Generative AI we Trust: Can Chatbots Effectively Verify Political Information?](https://mlsanity.com/paper/2312.13096/) (2023)
- [Towards Effective Discrimination Testing for Generative AI](https://mlsanity.com/paper/2412.21052/) (2024)
- [Gender Bias in Machine Translation and The Era of Large Language Models](https://mlsanity.com/paper/2401.10016/) (2024)
- [Is the U.S. Legal System Ready for AI's Challenges to Human Values?](https://mlsanity.com/paper/2308.15906/) (2023)
- [Transforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronouns](https://mlsanity.com/paper/2405.00134/) (2024)
- [A Taxonomy of Stereotype Content in Large Language Models](https://mlsanity.com/paper/2408.00162/) (2024)
- [Gender Representation and Bias in Indian Civil Service Mock Interviews](https://mlsanity.com/paper/2409.12194/) (2024)
- [Uncertainty Quantification for Evaluating Machine Translation Bias](https://mlsanity.com/paper/2507.18338/) (2025)
- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)
- [Modeling the Sacred: Considerations when Using Religious Texts in Natural Language Processing](https://mlsanity.com/paper/2404.14740/) (2024)
- [Exploring LGBTQ+ Bias in Generative AI Answers across Different Country and Religious Contexts](https://mlsanity.com/paper/2407.03473/) (2024)
- [Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religions](https://mlsanity.com/paper/2512.03943/) (2025)
- [Interrogating Design Homogenization in Web Vibe Coding](https://mlsanity.com/paper/2603.13036/) (2026)
- [Misgendering and Assuming Gender in Machine Translation when Working with Low-Resource Languages](https://mlsanity.com/paper/2401.13165/) (2024)
- [A Bayesian account of pronoun and neopronoun acquisition](https://mlsanity.com/paper/2504.02973/) (2025)

## Abstract (arXiv)

In this multicultural age, language translation is one of the most performed tasks, and it is becoming increasingly AI-moderated and automated. As a novel AI system, ChatGPT claims to be proficient in such translation tasks and in this paper, we put that claim to the test. Specifically, we examine ChatGPT's accuracy in translating between English and languages that exclusively use gender-neutral pronouns. We center this study around Bengali, the 7th most spoken language globally, but also generalize our findings across five other languages: Farsi, Malay, Tagalog, Thai, and Turkish. We find that ChatGPT perpetuates gender defaults and stereotypes assigned to certain occupations (e.g. man = doctor, woman = nurse) or actions (e.g. woman = cook, man = go to work), as it converts gender-neutral pronouns in languages to `he' or `she'. We also observe ChatGPT completely failing to translate the English gender-neutral pronoun `they' into equivalent gender-neutral pronouns in other languages, as it produces translations that are incoherent and incorrect. While it does respect and provide appropriately gender-marked versions of Bengali words when prompted with gender information in English, ChatGPT appears to confer a higher respect to men than to women in the same occupation. We conclude that ChatGPT exhibits the same gender biases which have been demonstrated for tools like Google Translate or MS Translator, as we provide recommendations for a human centered approach for future designers of AIs that perform language translation to better accommodate such low-resource languages.