# GATE: A Challenge Set for Gender-Ambiguous Translation Examples

arXiv: https://arxiv.org/abs/2303.03975 · HTML: https://mlsanity.com/paper/2303.03975/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus](https://mlsanity.com/paper/2006.05754/) (2020)
- [NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives](https://mlsanity.com/paper/2109.06105/) (2021)
- [They, Them, Theirs: Rewriting with Gender-Neutral English](https://mlsanity.com/paper/2102.06788/) (2021)
- [MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation](https://mlsanity.com/paper/2211.01355/) (2022)
- [Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translation](https://mlsanity.com/paper/2106.00169/) (2021)
- [Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation](https://mlsanity.com/paper/2203.09866/) (2022)
- [Investigating Failures of Automatic Translation in the Case of Unambiguous Gender](https://mlsanity.com/paper/2104.07838/) (2021)
- [GENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena](https://mlsanity.com/paper/2108.02854/) (2021)
- [Examining Covert Gender Bias: A Case Study in Turkish and English Machine Translation Models](https://mlsanity.com/paper/2108.10379/) (2021)
- [The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses](https://mlsanity.com/paper/2110.09216/) (2021)

## Built on by (most cited first, 18 of 18)

- [Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus](https://mlsanity.com/paper/2310.05294/) (2023)
- [Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling](https://mlsanity.com/paper/2311.18711/) (2023)
- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- [Enhancing Gender-Inclusive Machine Translation with Neomorphemes and Large Language Models](https://mlsanity.com/paper/2405.08477/) (2024)
- [Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INES](https://mlsanity.com/paper/2310.19345/) (2023)
- [Context-Aware or Context-Insensitive? Assessing LLMs' Performance in Document-Level Translation](https://mlsanity.com/paper/2410.14391/) (2024)
- [Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation](https://mlsanity.com/paper/2311.03767/) (2023)
- [GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages](https://mlsanity.com/paper/2402.14277/) (2024)
- [Generating Gender Alternatives in Machine Translation](https://mlsanity.com/paper/2407.20438/) (2024)
- [Good, but not always Fair: An Evaluation of Gender Bias for three commercial Machine Translation Systems](https://mlsanity.com/paper/2306.05882/) (2023)
- [On Instruction-Finetuning Neural Machine Translation Models](https://mlsanity.com/paper/2410.05553/) (2024)
- [Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations](https://mlsanity.com/paper/2506.00748/) (2025)
- [Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words](https://mlsanity.com/paper/2407.16266/) (2024)
- [FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimation](https://mlsanity.com/paper/2604.21420/) (2026)
- [Evaluating Gender Bias in the Translation of Gender-Neutral Languages into English](https://mlsanity.com/paper/2311.08836/) (2023)
- [Don't Overlook the Grammatical Gender: Bias Evaluation for Hindi-English Machine Translation](https://mlsanity.com/paper/2312.03710/) (2023)
- [Fine-grained Gender Control in Machine Translation with Large Language Models](https://mlsanity.com/paper/2407.15154/) (2024)
- [Explaining GAND: A Resource on Gender-Ambiguous Natural Data & Contrastive Attribution](https://mlsanity.com/paper/2607.22546/) (2026)

## Abstract (arXiv)

Although recent years have brought significant progress in improving translation of unambiguously gendered sentences, translation of ambiguously gendered input remains relatively unexplored. When source gender is ambiguous, machine translation models typically default to stereotypical gender roles, perpetuating harmful bias. Recent work has led to the development of "gender rewriters" that generate alternative gender translations on such ambiguous inputs, but such systems are plagued by poor linguistic coverage. To encourage better performance on this task we present and release GATE, a linguistically diverse corpus of gender-ambiguous source sentences along with multiple alternative target language translations. We also provide tools for evaluation and system analysis when using GATE and use them to evaluate our translation rewriter system.