# Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus

arXiv: https://arxiv.org/abs/2310.05294 · HTML: https://mlsanity.com/paper/2310.05294/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 8 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Toward Gender-Inclusive Coreference Resolution](https://mlsanity.com/paper/1910.13913/) (2019)
- [On Measuring Gender Bias in Translation of Gender-neutral Pronouns](https://mlsanity.com/paper/1905.11684/) (2019)
- [Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus](https://mlsanity.com/paper/2006.05754/) (2020)
- ["I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation](https://mlsanity.com/paper/2305.09941/) (2023)
- [Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender](https://mlsanity.com/paper/2202.11923/) (2022)
- [NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives](https://mlsanity.com/paper/2109.06105/) (2021)
- [They, Them, Theirs: Rewriting with Gender-Neutral English](https://mlsanity.com/paper/2102.06788/) (2021)
- [Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make It](https://mlsanity.com/paper/2010.05332/) (2020)
- [MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation](https://mlsanity.com/paper/2211.01355/) (2022)
- [ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages](https://mlsanity.com/paper/2305.10510/) (2023)
- [First the worst: Finding better gender translations during beam search](https://mlsanity.com/paper/2104.07429/) (2021)
- [Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation](https://mlsanity.com/paper/2203.09866/) (2022)
- [Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challenges](https://mlsanity.com/paper/2301.10075/) (2023)
- [Investigating Failures of Automatic Translation in the Case of Unambiguous Gender](https://mlsanity.com/paper/2104.07838/) (2021)
- [What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns](https://mlsanity.com/paper/2305.16051/) (2023)
- [GATE: A Challenge Set for Gender-Ambiguous Translation Examples](https://mlsanity.com/paper/2303.03975/) (2023)
- [GENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena](https://mlsanity.com/paper/2108.02854/) (2021)
- [How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronouns](https://mlsanity.com/paper/2204.10281/) (2022)
- [Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Model](https://mlsanity.com/paper/2305.11140/) (2023)
- [User-Centric Gender Rewriting](https://mlsanity.com/paper/2205.02211/) (2022)

## Built on by (most cited first, 20 of 20)

- [Gender Bias in Large Language Models across Multiple Languages](https://mlsanity.com/paper/2403.00277/) (2024)
- [A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation](https://mlsanity.com/paper/2310.12127/) (2023)
- [What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study](https://mlsanity.com/paper/2410.00545/) (2024)
- [A Prompt Response to the Demand for Automatic Gender-Neutral Translation](https://mlsanity.com/paper/2402.06041/) (2024)
- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- [Enhancing Gender-Inclusive Machine Translation with Neomorphemes and Large Language Models](https://mlsanity.com/paper/2405.08477/) (2024)
- [A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applications](https://mlsanity.com/paper/2503.20302/) (2025)
- [Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMs](https://mlsanity.com/paper/2409.13852/) (2024)
- [Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms](https://mlsanity.com/paper/2503.04372/) (2025)
- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metrics](https://mlsanity.com/paper/2510.06841/) (2025)
- [Generating Gender Alternatives in Machine Translation](https://mlsanity.com/paper/2407.20438/) (2024)
- [Gender-Neutral Machine Translation Strategies in Practice](https://mlsanity.com/paper/2506.15676/) (2025)
- [An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation](https://mlsanity.com/paper/2504.11934/) (2025)
- [Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations](https://mlsanity.com/paper/2506.00748/) (2025)
- [WMT24 Test Suite: Gender Resolution in Speaker-Listener Dialogue Roles](https://mlsanity.com/paper/2411.06194/) (2024)
- [GeNRe: A French Gender-Neutral Rewriting System Using Collective Nouns](https://mlsanity.com/paper/2505.23630/) (2025)
- [Fine-grained Gender Control in Machine Translation with Large Language Models](https://mlsanity.com/paper/2407.15154/) (2024)
- [Gender-Neutral Rewriting in Italian: Models, Approaches, and Trade-offs](https://mlsanity.com/paper/2509.13480/) (2025)
- [Explaining GAND: A Resource on Gender-Ambiguous Natural Data & Contrastive Attribution](https://mlsanity.com/paper/2607.22546/) (2026)

## Abstract (arXiv)

Gender inequality is embedded in our communication practices and perpetuated in translation technologies. This becomes particularly apparent when translating into grammatical gender languages, where machine translation (MT) often defaults to masculine and stereotypical representations by making undue binary gender assumptions. Our work addresses the rising demand for inclusive language by focusing head-on on gender-neutral translation from English to Italian. We start from the essentials: proposing a dedicated benchmark and exploring automated evaluation methods. First, we introduce GeNTE, a natural, bilingual test set for gender-neutral translation, whose creation was informed by a survey on the perception and use of neutral language. Based on GeNTE, we then overview existing reference-based evaluation approaches, highlight their limits, and propose a reference-free method more suitable to assess gender-neutral translation.