Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructure
Not on the map. Why?
Abstract (arXiv)
We present Racka, a lightweight, continually pretrained large language model designed to bridge the resource gap between Hungarian and high-resource languages such as English and German. Racka employs parameter-efficient continual pretraining via Low-Rank Adaptation (LoRA) on a Qwen-3 4B backbone, making the recipe practical on A100 (40GB)-based HPC clusters with low inter-node bandwidth. To better match the training distribution, we replace and adapt the tokenizer, achieving substantially improved tokenization fertility for Hungarian while maintaining competitive performance in English and German. The model is trained on 160B subword tokens drawn from a mixture of internet and high-quality curated sources, with a composition of 44% Hungarian, 24% English, 21% German, and 11% code. This data mix is chosen to mitigate catastrophic forgetting and preserve high-resource language capabilities during continual pretraining. Our preliminary results indicate modest but stable results in language adaptation.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 3 Jan 2026.
Builds on
21 of 37 references.
References · 21
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
- 2023Continual Pre-training of Language Modelscited by 194
- 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
- 2024EuroLLM: Multilingual Language Models for Europecited by 103
- 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
- 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
- 2024Towards Multilingual LLM Evaluation for European Languagescited by 29
9 more
- 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
- 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Modelscited by 16
- 2021Fine-Tuning Transformers: Vocabulary Transfercited by 14
- 2024Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuningcited by 14
- 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
- 2021Evaluating Transferability of BERT Models on Uralic Languagescited by 6
- 2024A Survey of Large Language Models for European Languagescited by 6
- 2024GEITje 7B Ultra: A Conversational Model for Dutchcited by 4
- 2025OpenHuEval: Evaluating Large Language Model on Hungarian Specificscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.