Kakugo: Distillation of Low-Resource Languages into Small Language Models
Not on the map. Why?
Abstract (arXiv)
We present Kakugo, a novel and cost-effective pipeline designed to train general-purpose Small Language Models (SLMs) for low-resource languages using only the language name as input. By using a large teacher model to generate synthetic prompts and translate instruction datasets, we produced training data and SLMs for 54 low-resource languages. Evaluations across a diverse set of general natural language processing tasks, including translation, classification, and question answering, demonstrate that our pipeline consistently improves performance over base models. With a total generation and training cost of under 50 per language, Kakugo offers an accessible method for communities to develop language-specific AI.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 58 references that are in the map. First posted on arXiv 20 Jan 2026.
Builds on
32 of 58 references.
References · 32
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
- 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
- 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
- 2023Scaling Speech Technology to 1,000+ Languagescited by 411
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
20 more
- 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
- 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
- 2021MuRIL: Multilingual Representations for Indian Languagescited by 179
- 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
- 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
- 2021Towards Zero-Label Language Learningcited by 76
- 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
- 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 50
- 2025Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Modelscited by 50
- 2025Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languagescited by 34
- 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
- 2024DiaSynth: Synthetic Dialogue Generation Framework for Low Resource Dialogue Applicationscited by 18
- 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
- 2023DALE: Generative Data Augmentation for Low-Resource Legal NLPcited by 12
- 2024MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructionscited by 12
- 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Datacited by 12
- 2024LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexiconscited by 11
- 2024SlimLM: An Efficient Small Language Model for On-Device Document Assistancecited by 9
- 2024Scaling A Simple Approach to Zero-Shot Speech Recognitioncited by 8
- 2025Beyond Translation: LLM-Based Data Generation for Multilingual Fact-Checkingcited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.