# What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study

arXiv: https://arxiv.org/abs/2410.00545 · HTML: https://mlsanity.com/paper/2410.00545/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 164 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 30 of 75)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://mlsanity.com/paper/2012.15859/) (2020)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Getting Gender Right in Neural Machine Translation](https://mlsanity.com/paper/1909.05088/) (2019)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem](https://mlsanity.com/paper/2004.04498/) (2020)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations](https://mlsanity.com/paper/2203.13928/) (2022)
- [Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation](https://mlsanity.com/paper/2102.00287/) (2021)
- [Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques](https://mlsanity.com/paper/1901.03116/) (2019)
- [Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation](https://mlsanity.com/paper/2109.03858/) (2021)
- [On Measuring Gender Bias in Translation of Gender-neutral Pronouns](https://mlsanity.com/paper/1905.11684/) (2019)
- [Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus](https://mlsanity.com/paper/2006.05754/) (2020)
- [Rethinking Model Evaluation as Narrowing the Socio-Technical Gap](https://mlsanity.com/paper/2306.03100/) (2023)
- [Personalized Machine Translation: Preserving Original Author Traits](https://mlsanity.com/paper/1610.05461/) (2016)
- [NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives](https://mlsanity.com/paper/2109.06105/) (2021)
- [Mitigating Gender Bias in Machine Translation with Target Gender Annotations](https://mlsanity.com/paper/2010.06203/) (2020)
- [Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make It](https://mlsanity.com/paper/2010.05332/) (2020)
- [MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation](https://mlsanity.com/paper/2211.01355/) (2022)
- [DEMETR: Diagnosing Evaluation Metrics for Translation](https://mlsanity.com/paper/2210.13746/) (2022)
- [First the worst: Finding better gender translations during beam search](https://mlsanity.com/paper/2104.07429/) (2021)
- [Choose Your Lenses: Flaws in Gender Bias Evaluation](https://mlsanity.com/paper/2210.11471/) (2022)
- [Automatically Identifying Gender Issues in Machine Translation using Perturbations](https://mlsanity.com/paper/2004.14065/) (2020)

## Built on by (most cited first, 15 of 15)

- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- [Translation in the Hands of Many:Centering Lay Users in Machine Translation Interactions](https://mlsanity.com/paper/2502.13780/) (2025)
- [An Interdisciplinary Approach to Human-Centered Machine Translation](https://mlsanity.com/paper/2506.13468/) (2025)
- [Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms](https://mlsanity.com/paper/2503.04372/) (2025)
- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metrics](https://mlsanity.com/paper/2510.06841/) (2025)
- [Exploring the Translation Mechanism of Large Language Models](https://mlsanity.com/paper/2502.11806/) (2025)
- [Gender-Neutral Machine Translation Strategies in Practice](https://mlsanity.com/paper/2506.15676/) (2025)
- [An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation](https://mlsanity.com/paper/2504.11934/) (2025)
- [FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimation](https://mlsanity.com/paper/2604.21420/) (2026)
- [MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation](https://mlsanity.com/paper/2412.11615/) (2024)
- [Gender-Neutral Rewriting in Italian: Models, Approaches, and Trade-offs](https://mlsanity.com/paper/2509.13480/) (2025)
- [Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lens](https://mlsanity.com/paper/2511.03880/) (2025)
- [Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios](https://mlsanity.com/paper/2606.06177/) (2026)
- [Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations](https://mlsanity.com/paper/2609.21490/) (2026)

## Abstract (arXiv)

Gender bias in machine translation (MT) is recognized as an issue that can harm people and society. And yet, advancements in the field rarely involve people, the final MT users, or inform how they might be impacted by biased technologies. Current evaluations are often restricted to automatic methods, which offer an opaque estimate of what the downstream impact of gender disparities might be. We conduct an extensive human-centered study to examine if and to what extent bias in MT brings harms with tangible costs, such as quality of service gaps across women and men. To this aim, we collect behavioral data from 90 participants, who post-edited MT outputs to ensure correct gender translation. Across multiple datasets, languages, and types of users, our study shows that feminine post-editing demands significantly more technical and temporal effort, also corresponding to higher financial costs. Existing bias measurements, however, fail to reflect the found disparities. Our findings advocate for human-centered approaches that can inform the societal impact of bias.