# EuroGEST: Investigating gender stereotypes in multilingual language models

arXiv: https://arxiv.org/abs/2506.03867 · HTML: https://mlsanity.com/paper/2506.03867/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 56 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Evaluating and Mitigating Discrimination in Language Model Decisions](https://mlsanity.com/paper/2312.03689/) (2023)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://mlsanity.com/paper/2404.05399/) (2024)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus](https://mlsanity.com/paper/2006.05754/) (2020)
- [Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias](https://mlsanity.com/paper/2206.09860/) (2022)
- [Building Socio-culturally Inclusive Stereotype Resources with Community Engagement](https://mlsanity.com/paper/2307.10514/) (2023)
- [CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias](https://mlsanity.com/paper/2308.12539/) (2023)
- [Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling](https://mlsanity.com/paper/2311.18711/) (2023)
- [Public Perceptions of Gender Bias in Large Language Models: Cases of ChatGPT and Ernie](https://mlsanity.com/paper/2309.09120/) (2023)

## Built on by (most cited first, 4 of 4)

- [Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback](https://mlsanity.com/paper/2511.08225/) (2025)
- [Teaching and Critiquing Conceptualization and Operationalization in NLP](https://mlsanity.com/paper/2512.18505/) (2025)
- [Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts](https://mlsanity.com/paper/2606.30152/) (2026)
- [Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Information](https://mlsanity.com/paper/2610.01696/) (2026)

## Abstract (arXiv)

Large language models increasingly support multiple languages, yet most benchmarks for gender bias remain English-centric. We introduce EuroGEST, a dataset designed to measure gender-stereotypical reasoning in LLMs across English and 29 European languages. EuroGEST builds on an existing expert-informed benchmark covering 16 gender stereotypes, expanded in this work using translation tools, quality estimation metrics, and morphological heuristics. Human evaluations confirm that our data generation method results in high accuracy of both translations and gender labels across languages. We use EuroGEST to evaluate 24 multilingual language models from six model families, demonstrating that the strongest stereotypes in all models across all languages are that women are 'beautiful', 'empathetic' and 'neat' and men are 'leaders', 'strong, tough' and 'professional'. We also show that larger models encode gendered stereotypes more strongly and that instruction finetuning does not consistently reduce gendered stereotypes. Our work highlights the need for more multilingual studies of fairness in LLMs and offers scalable methods and resources to audit gender bias across languages.