# Automatically Identifying Gender Issues in Machine Translation using Perturbations

arXiv: https://arxiv.org/abs/2004.14065 · HTML: https://mlsanity.com/paper/2004.14065/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 28 of 28)

- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation](https://mlsanity.com/paper/2109.03858/) (2021)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Investigating Failures of Automatic Translation in the Case of Unambiguous Gender](https://mlsanity.com/paper/2104.07838/) (2021)
- [What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns](https://mlsanity.com/paper/2305.16051/) (2023)
- [A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation](https://mlsanity.com/paper/2310.12127/) (2023)
- [Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models](https://mlsanity.com/paper/2408.06518/) (2024)
- [What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study](https://mlsanity.com/paper/2410.00545/) (2024)
- [The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses](https://mlsanity.com/paper/2110.09216/) (2021)
- [How to Split: the Effect of Word Segmentation on Gender Bias in Speech Translation](https://mlsanity.com/paper/2105.13782/) (2021)
- [How to Write a Bias Statement: Recommendations for Submissions to the Workshop on Gender Bias in NLP](https://mlsanity.com/paper/2104.03026/) (2021)
- [Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms](https://mlsanity.com/paper/2503.04372/) (2025)
- [Controlling Extra-Textual Attributes about Dialogue Participants -- A Case Study of English-to-Polish Neural Machine Translation](https://mlsanity.com/paper/2205.04747/) (2022)
- [That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?](https://mlsanity.com/paper/2310.14610/) (2023)
- [Gender Bias in English-to-Greek Machine Translation](https://mlsanity.com/paper/2506.09558/) (2025)
- [Uncertainty Quantification for Evaluating Machine Translation Bias](https://mlsanity.com/paper/2507.18338/) (2025)
- [From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text](https://mlsanity.com/paper/2508.09450/) (2025)
- [Screening Gender Transfer in Neural Machine Translation](https://mlsanity.com/paper/2202.12568/) (2022)
- [The User-Aware Arabic Gender Rewriter](https://mlsanity.com/paper/2210.07538/) (2022)
- [Investigating Cross-Linguistic Gender Bias in Hindi-English Across Domains](https://mlsanity.com/paper/2111.11159/) (2021)

## Abstract (arXiv)

The successful application of neural methods to machine translation has realized huge quality advances for the community. With these improvements, many have noted outstanding challenges, including the modeling and treatment of gendered language. While previous studies have identified issues using synthetic examples, we develop a novel technique to mine examples from real world data to explore challenges for deployed systems. We use our method to compile an evaluation benchmark spanning examples for four languages from three language families, which we publicly release to facilitate research. The examples in our benchmark expose where model representations are gendered, and the unintended consequences these gendered representations can have in downstream application.