# Gender Bias in MT for a Genderless Language: New Benchmarks for Basque

arXiv: https://arxiv.org/abs/2603.08153 · HTML: https://mlsanity.com/paper/2603.08153/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 16 of 16)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Getting Gender Right in Neural Machine Translation](https://mlsanity.com/paper/1909.05088/) (2019)
- [Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias](https://mlsanity.com/paper/2010.14534/) (2020)
- [KoBBQ: Korean Bias Benchmark for Question Answering](https://mlsanity.com/paper/2307.16778/) (2023)
- [Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation](https://mlsanity.com/paper/2109.03858/) (2021)
- [The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations](https://mlsanity.com/paper/2308.02053/) (2023)
- [Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale](https://mlsanity.com/paper/2305.13198/) (2023)
- [Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms](https://mlsanity.com/paper/2503.04372/) (2025)
- [GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages](https://mlsanity.com/paper/2402.14277/) (2024)
- [Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations](https://mlsanity.com/paper/2506.00748/) (2025)

## Abstract (arXiv)

Large language models (LLMs) and machine translation (MT) systems are increasingly used in our daily lives, but their outputs can reproduce gender bias present in the training data. Most resources for evaluating such biases are designed for English and reflect its sociocultural context, which limits their applicability to other languages. This work addresses this gap by introducing two new datasets to evaluate gender bias in translations involving Basque, a low-resource and genderless language. WinoMTeus adapts the WinoMT benchmark to examine how gender-neutral Basque occupations are translated into gendered languages such as Spanish and French. FLORES+Gender, in turn, extends the FLORES+ benchmark to assess whether translation quality varies when translating from gendered languages (Spanish and English) into Basque depending on the gender of the referent. We evaluate several general-purpose LLMs and open and proprietary MT systems. The results reveal a systematic preference for masculine forms and, in some models, a slightly higher quality for masculine referents. Overall, these findings show that gender bias is still deeply rooted in these models, and highlight the need to develop evaluation methods that consider both linguistic features and cultural context.