DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer
Not on the map. Why?
Abstract (arXiv)
Small language models (SLMs) are efficient and scalable, but their multilingual capabilities degrade severely at sub-billion scales, especially for Southeast Asian (SEA) languages. We introduce DuDi, a dual-signal multilingual distillation framework that combines an online sequence-level signal with off-policy and on-policy token-level signals. DuDi further uses a cross-lingual verbalizer to refine teacher feedback and improve teacher-student transferability in multilingual settings. Experiments on SEA-HELM across multiple model families, scales, and teacher-student settings show that DuDi consistently outperforms competitive distillation baselines. Ablations and analyses confirm that sequence-level optimization, token-level supervision, and cross-lingual verbalization provide complementary and transferable learning signals for multilingual SLMs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 36 references that are in the map. First posted on arXiv 3 Jun 2026.
Builds on
18 of 36 references.
References · 18
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
- 2016Sequence-Level Knowledge Distillationcited by 761
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
- 2026Reinforcement Learning via Self-Distillationcited by 285
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
- 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
- 2026Self-Distillation Enables Continual Learningcited by 212
- 2024How do Large Language Models Handle Multilingualism?cited by 148
- 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
- 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
- 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
6 more
- 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
- 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 37
- 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
- 2017An optimal transportation approach for assessing almost stochastic ordercited by 33
- 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasonerscited by 16
- 2024SlimLM: An Efficient Small Language Model for On-Device Document Assistancecited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.