# Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs

arXiv: https://arxiv.org/abs/2410.03730 · HTML: https://mlsanity.com/paper/2410.03730/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 78 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)

## Built on by (most cited first, 22 of 22)

- [Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study](https://mlsanity.com/paper/2505.06149/) (2025)
- [IberBench: LLM Evaluation on Iberian Languages](https://mlsanity.com/paper/2504.16921/) (2025)
- [DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing](https://mlsanity.com/paper/2504.21589/) (2025)
- [Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian](https://mlsanity.com/paper/2509.05668/) (2025)
- [Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness](https://mlsanity.com/paper/2510.06780/) (2025)
- [Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation](https://mlsanity.com/paper/2608.25089/) (2026)
- [Experiments with Large Language Models on Retrieval-Augmented Generation for Closed-Source Simulation Software](https://mlsanity.com/paper/2502.03916/) (2025)
- [MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP](https://mlsanity.com/paper/2506.04385/) (2025)
- [La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America](https://mlsanity.com/paper/2507.00999/) (2025)
- [TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation](https://mlsanity.com/paper/2603.08182/) (2026)
- [XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity](https://mlsanity.com/paper/2605.05662/) (2026)
- [Are Multilingual Language Models an Off-ramp for Under-resourced Languages? Will we arrive at Digital Language Equality in Europe in 2030?](https://mlsanity.com/paper/2502.12886/) (2025)
- [Multilingual European Language Models: Benchmarking Approaches and Challenges](https://mlsanity.com/paper/2502.12895/) (2025)
- [A Cartography of Open Collaboration in Open Source AI: Mapping Practices, Motivations, and Governance in 14 Open Large Language Model Projects](https://mlsanity.com/paper/2509.25397/) (2025)
- [KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report](https://mlsanity.com/paper/2606.03773/) (2026)
- [A Sovereign, Open-Source Foundation Model for German and English](https://mlsanity.com/paper/2607.09424/) (2026)
- [Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs](https://mlsanity.com/paper/2609.07664/) (2026)
- [Security and Detectability Analysis of Unicode Text Watermarking Methods against Large Language Models](https://mlsanity.com/paper/2512.13325/) (2025)
- [BYOL: Bring Your Own Language Into LLMs](https://mlsanity.com/paper/2601.10804/) (2026)
- [Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset](https://mlsanity.com/paper/2603.24772/) (2026)
- [Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation](https://mlsanity.com/paper/2609.30009/) (2026)
- [sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak](https://mlsanity.com/paper/2610.09152/) (2026)

## Abstract (arXiv)

We present two multilingual LLMs, Teuken 7B-base and Teuken 7B-instruct, designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. Trained on a dataset comprising around 60% non-English data and utilizing a custom multilingual tokenizer, our models address the limitations of existing LLMs that predominantly focus on English or a few high-resource languages. We detail the models' development principles, i.e., data composition, tokenizer optimization, and training methodologies. The models demonstrate strong performance across multilingual benchmarks, as evidenced by their performance on European versions of ARC, HellaSwag, and TruthfulQA.