mlsanityData to 14 Oct 2026

Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs

arXiv 2410.03730 ↗First posted 30 Sep 2024cs.CL cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

We present two multilingual LLMs, Teuken 7B-base and Teuken 7B-instruct, designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. Trained on a dataset comprising around 60% non-English data and utilizing a custom multilingual tokenizer, our models address the limitations of existing LLMs that predominantly focus on English or a few high-resource languages. We detail the models' development principles, i.e., data composition, tokenizer optimization, and training methodologies. The models demonstrate strong performance across multilingual benchmarks, as evidenced by their performance on European versions of ARC, HellaSwag, and TruthfulQA.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 51 of its 78 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

51 of 78 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  10. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,963
38 more
  1. 2016Layer Normalizationcited by 5,669
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023Mistral 7Bcited by 3,164
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  12. 2020GLU Variants Improve Transformercited by 1,804
  13. 2019Root Mean Square Layer Normalizationcited by 1,480
  14. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  15. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  16. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  17. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  18. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  19. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  20. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  21. 2016Language Modeling with Gated Convolutional Networkscited by 702
  22. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  23. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  24. 2023Textbooks Are All You Needcited by 569
  25. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  26. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  27. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  28. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  29. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  30. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  31. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  32. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  33. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  34. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  35. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 100
  36. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  37. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  38. 2024PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Modelscited by 20

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Studycited by 7
  2. 2025IberBench: LLM Evaluation on Iberian Languagescited by 5
  3. 2025DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexingcited by 4
  4. 2025Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavariancited by 4
  5. 2025Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustnesscited by 3
  6. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  7. 2025Experiments with Large Language Models on Retrieval-Augmented Generation for Closed-Source Simulation Softwarecited by 2
  8. 2025MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLPcited by 2
  9. 2025La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin Americacited by 2
  10. 2026TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representationcited by 2
  11. 2026XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivitycited by 2
  12. 2025Are Multilingual Language Models an Off-ramp for Under-resourced Languages? Will we arrive at Digital Language Equality in Europe in 2030?cited by 1
10 more
  1. 2025Multilingual European Language Models: Benchmarking Approaches and Challengescited by 1
  2. 2025A Cartography of Open Collaboration in Open Source AI: Mapping Practices, Motivations, and Governance in 14 Open Large Language Model Projectscited by 1
  3. 2026KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Reportcited by 1
  4. 2026A Sovereign, Open-Source Foundation Model for German and Englishcited by 1
  5. 2026Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMscited by 1
  6. 2025Security and Detectability Analysis of Unicode Text Watermarking Methods against Large Language Modelscited by 0
  7. 2026BYOL: Bring Your Own Language Into LLMscited by 0
  8. 2026Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Datasetcited by 0
  9. 2026Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generationcited by 0
  10. 2026sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovakcited by 0