# EuroLLM: Multilingual Language Models for Europe

arXiv: https://arxiv.org/abs/2409.16235 · HTML: https://mlsanity.com/paper/2409.16235/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [On the Use of ArXiv as a Dataset](https://mlsanity.com/paper/1905.00075/) (2019)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [A Walk with SGD](https://mlsanity.com/paper/1802.08770/) (2018)
- [Scaling Laws for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2302.09650/) (2023)

## Built on by (most cited first, 30 of 106)

- [LEXam: Benchmarking Legal Reasoning on 340 Law Exams](https://mlsanity.com/paper/2505.12864/) (2025)
- [mmBERT: A Modern Multilingual Encoder with Annealed Language Learning](https://mlsanity.com/paper/2509.06888/) (2025)
- [Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs](https://mlsanity.com/paper/2410.03730/) (2024)
- [BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation](https://mlsanity.com/paper/2502.04314/) (2025)
- [Granary: Speech Recognition and Translation Dataset in 25 European Languages](https://mlsanity.com/paper/2505.13404/) (2025)
- [Localized Cultural Knowledge is Conserved and Controllable in Large Language Models](https://mlsanity.com/paper/2504.10191/) (2025)
- [Multilingual Language Model Pretraining using Machine-translated Data](https://mlsanity.com/paper/2502.13252/) (2025)
- [Krikri: Advancing Open Large Language Models for Greek](https://mlsanity.com/paper/2505.13772/) (2025)
- [The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technology](https://mlsanity.com/paper/2506.01340/) (2025)
- [Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque](https://mlsanity.com/paper/2506.07597/) (2025)
- [Multilingual Contextualization of Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2504.12140/) (2025)
- [Beyond Text Compression: Evaluating Tokenizers Across Scales](https://mlsanity.com/paper/2506.03101/) (2025)
- [LL"aMmlein: Transparent, Compact and Competitive German-Only Language Models from Scratch](https://mlsanity.com/paper/2411.11171/) (2024)
- [TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2506.13487/) (2025)
- [Gaperon: A Peppered English-French Generative Language Model Suite](https://mlsanity.com/paper/2510.25771/) (2025)
- [Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark](https://mlsanity.com/paper/2502.07057/) (2025)
- [From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM](https://mlsanity.com/paper/2503.10620/) (2025)
- [Minimal Pair-Based Evaluation of Code-Switching](https://mlsanity.com/paper/2506.01840/) (2025)
- [Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025](https://mlsanity.com/paper/2506.17077/) (2025)
- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)
- [Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages](https://mlsanity.com/paper/2508.05149/) (2025)
- [Tokens with Meaning: A Hybrid Tokenization Approach for Turkish](https://mlsanity.com/paper/2508.14292/) (2025)
- [Context-Aware or Context-Insensitive? Assessing LLMs' Performance in Document-Level Translation](https://mlsanity.com/paper/2410.14391/) (2024)
- [Training-free LLM Merging for Multi-task Learning](https://mlsanity.com/paper/2506.12379/) (2025)
- [Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison](https://mlsanity.com/paper/2509.09009/) (2025)
- [Extending Automatic Machine Translation Evaluation to Book-Length Documents](https://mlsanity.com/paper/2509.17249/) (2025)
- [Bielik 11B v3: Multilingual Large Language Model for European Languages](https://mlsanity.com/paper/2601.11579/) (2025)
- [GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek](https://mlsanity.com/paper/2602.05150/) (2026)
- [From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages](https://mlsanity.com/paper/2410.18836/) (2024)
- [Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language](https://mlsanity.com/paper/2410.23956/) (2024)

## Abstract (arXiv)

The quality of open-weight LLMs has seen significant improvement, yet they remain predominantly focused on English. In this paper, we introduce the EuroLLM project, aimed at developing a suite of open-weight multilingual LLMs capable of understanding and generating text in all official European Union languages, as well as several additional relevant languages. We outline the progress made to date, detailing our data collection and filtering process, the development of scaling laws, the creation of our multilingual tokenizer, and the data mix and modeling configurations. Additionally, we release our initial models: EuroLLM-1.7B and EuroLLM-1.7B-Instruct and report their performance on multilingual general benchmarks and machine translation.