Good, but not always Fair: An Evaluation of Gender Bias for three commercial Machine Translation Systems
Not on the map. Why?
Abstract (arXiv)
Machine Translation (MT) continues to make significant strides in quality and is increasingly adopted on a larger scale. Consequently, analyses have been redirected to more nuanced aspects, intricate phenomena, as well as potential risks that may arise from the widespread use of MT tools. Along this line, this paper offers a meticulous assessment of three commercial MT systems - Google Translate, DeepL, and Modern MT - with a specific focus on gender translation and bias. For three language pairs (English/Spanish, English/Italian, and English/French), we scrutinize the behavior of such systems at several levels of granularity and on a variety of naturally occurring gender phenomena in translation. Our study takes stock of the current state of online MT tools, by revealing significant discrepancies in the gender translation of the three systems, with each system displaying varying degrees of bias despite their overall translation quality.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 58 references that are in the map. First posted on arXiv 9 Jun 2023.
Builds on
25 of 58 references.
References · 25
- 2017Attention Is All You Needcited by 60,361
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
- 2015Attention-Based Models for Speech Recognitioncited by 401
- 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
- 2019Evaluating Gender Bias in Machine Translationcited by 249
- 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
- 2018Assessing Gender Bias in Machine Translation -- A Case Study with Google Translatecited by 121
- 2021Gender Bias in Machine Translationcited by 117
- 2019Getting Gender Right in Neural Machine Translationcited by 98
13 more
- 2020Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problemcited by 86
- 2019Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniquescited by 65
- 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
- 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
- 2020Mitigating Gender Bias in Machine Translation with Target Gender Annotationscited by 42
- 2022MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translationcited by 38
- 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
- 2021Investigating Failures of Automatic Translation in the Case of Unambiguous Gendercited by 20
- 2016Neural versus Phrase-Based Machine Translation Quality: a Case Studycited by 19
- 2023GATE: A Challenge Set for Gender-Ambiguous Translation Examplescited by 18
- 2020Breeding Gender-aware Direct Speech Translation Systemscited by 16
- 2017A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directionscited by 10
- 2022How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contextscited by 10
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.