# Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network

arXiv: https://arxiv.org/abs/2607.18432 · HTML: https://mlsanity.com/paper/2607.18432/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 12 references that are in the map.
- First posted on arXiv 20 Jul 2026.

## Builds on (most cited first, 3 of 3)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [A Call for More Rigor in Unsupervised Cross-lingual Learning](https://mlsanity.com/paper/2004.14958/) (2020)
- [Are Multilingual Language Models an Off-ramp for Under-resourced Languages? Will we arrive at Digital Language Equality in Europe in 2030?](https://mlsanity.com/paper/2502.12886/) (2025)

## Abstract (arXiv)

This paper reports on a collaboration between the Directorate-General for Translation (DGT) and the European Master's in Translation (EMT) to localise the MMLU dataset into 11 European languages. Beyond creating a more inclusive benchmark for LLM evaluation, the project offers master's students authentic, project-based professional training in translation, revision, project management, and multilingual coordination, while highlighting key methodological, administrative, and workflow challenges.