# Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian

arXiv: https://arxiv.org/abs/2509.05668 · HTML: https://mlsanity.com/paper/2509.05668/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 6 Sep 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://mlsanity.com/paper/2310.08754/) (2023)
- [LLaMA Pro: Progressive LLaMA with Block Expansion](https://mlsanity.com/paper/2401.02415/) (2024)
- [PolyLM: An Open Source Polyglot Large Language Model](https://mlsanity.com/paper/2307.06018/) (2023)
- [LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback](https://mlsanity.com/paper/2406.01771/) (2024)
- [Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs](https://mlsanity.com/paper/2410.03730/) (2024)
- [EuroLLM-9B: Technical Report](https://mlsanity.com/paper/2506.04079/) (2025)
- [Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi](https://mlsanity.com/paper/2504.06011/) (2025)
- [MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank](https://mlsanity.com/paper/2403.10293/) (2024)
- [A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation](https://mlsanity.com/paper/2506.02894/) (2025)
- [Bilingual Adaptation of Monolingual Foundation Models](https://mlsanity.com/paper/2407.12869/) (2024)

## Built on by (most cited first, 4 of 4)

- [Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models](https://mlsanity.com/paper/2602.08658/) (2026)
- [BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian](https://mlsanity.com/paper/2608.12894/) (2026)
- [Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling](https://mlsanity.com/paper/2604.28075/) (2026)
- [Lightweight but Linguistically Fair? Quantisation Robustness across Bavarian, German, and English](https://mlsanity.com/paper/2610.18343/) (2026)

## Abstract (arXiv)

We present Llama-GENBA-10B, a trilingual foundation model addressing English-centric bias in large language models. Built on Llama 3.1-8B and scaled to 10B parameters, Llama-GENBA-10B is continuously pretrained on 164B tokens (82B English, 82B German, and 80M Bavarian), balancing resources while preventing English dominance. Targeted at the German NLP community, the model also promotes Bavarian as a low-resource language. Development tackled four challenges: (1) curating a multilingual corpus despite Bavarian scarcity, (2) creating a unified tokenizer for English, German, and Bavarian, (3) optimizing architecture and language-ratio hyperparameters for cross-lingual transfer, and (4) establishing the first standardized trilingual evaluation suite by translating German benchmarks into Bavarian. Evaluations show that Llama-GENBA-10B achieves strong cross-lingual performance, with the fine-tuned variant surpassing Apertus-8B-2509 and gemma-2-9b in Bavarian and establishing itself as the best model in its class for this language, while also outperforming EuroLLM in English and matching its results in German. Training on the Cerebras CS-2 demonstrated efficient large-scale multilingual pretraining with documented energy use, offering a blueprint for inclusive foundation models that integrate low-resource languages.