mlsanityData to 14 Oct 2026

Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages

arXiv 2402.12204 ↗First posted 19 Feb 2024cs.CLcited by 36

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

While large language models (LLMs) have been pre-trained on multilingual corpora, their performance still lags behind in most languages compared to a few resource-rich languages. One common approach to mitigate this issue is to translate training data from resource-rich languages into other languages and then continue training. However, using the data obtained solely relying on translation while ignoring the original capabilities of LLMs across languages is not always effective, which we show will limit the performance of cross-lingual knowledge transfer. In this work, we propose SDRRL, a method based on Self-Distillation from Resource-Rich Languages that effectively improve multilingual performance by leveraging the internal capabilities of LLMs on resource-rich languages. We evaluate on different LLMs (LLaMA-2 and SeaLLM) and source languages across various comprehension and generation tasks, experimental results demonstrate that SDRRL can significantly enhance multilingual capabilities while minimizing the impact on original performance in resource-rich languages.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 39 of its 71 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

39 of 71 references.

References · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2020Knowledge Distillation: A Surveycited by 1,603
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  9. 2016Sequence-Level Knowledge Distillationcited by 801
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  11. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  12. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
27 more
  1. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  2. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  3. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  4. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  5. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 153
  6. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  7. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 97
  8. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  9. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  10. 2023Do Multilingual Language Models Think Better in English?cited by 88
  11. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  12. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 70
  13. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  14. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  15. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  16. 2024xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoningcited by 46
  17. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  18. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  19. 2022Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Promptcited by 25
  20. 2020Knowledge Distillation for Multilingual Unsupervised Neural Machine Translationcited by 19
  21. 2023Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddingscited by 19
  22. 2019Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translationcited by 16
  23. 2021WeChat Neural Machine Translation Systems for WMT21cited by 11
  24. 2021DirectQuote: A Dataset for Direct Quotation Extraction and Attribution in News Articlescited by 10
  25. 2023Empowering Multi-step Reasoning across Languages via Tree-of-Thoughtscited by 8
  26. 2023Several categories of Large Language Models (LLMs): A Short Surveycited by 7
  27. 2023AdvFusion: Adapter-based Knowledge Transfer for Code Summarization on Code Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 13.

36 papers.

Sort:

By area:

  1. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 28
  2. 2025Investigating Language Preference of Multilingual RAG Systemscited by 22
  3. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  4. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  5. 2025AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thoughtcited by 14
  6. 2024Learn and Unlearn: Addressing Misinformation in Multilingual LLMscited by 12
  7. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 12
  8. 2025Enhancing Non-English Capabilities of English-Centric Large Language Models through Deep Supervision Fine-Tuningcited by 11
  9. 2025Improving Multilingual Language Models by Aligning Representations through Steeringcited by 11
  10. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steeringcited by 10
  11. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  12. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
24 more
  1. 2025Just Go Parallel: Improving the Multilingual Capabilities of Large Language Modelscited by 8
  2. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 7
  3. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 6
  4. 2025AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignmentcited by 6
  5. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoningcited by 6
  6. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 5
  7. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 5
  8. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  9. 2026Cross-Lingual Activation Steering for Multilingual Language Modelscited by 5
  10. 2025Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-trainingcited by 4
  11. 2024LLMs for Extremely Low-Resource Finno-Ugric Languagescited by 3
  12. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 3
  13. 2026LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safetycited by 3
  14. 2026Multilingual Safety Alignment via Self-Distillationcited by 3
  15. 2026Layer-wise Swapping for Generalizable Multilingual Safetycited by 2
  16. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  17. 2026Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilingualitycited by 2
  18. 2026I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimizationcited by 2
  19. 2025CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuningcited by 1
  20. 2026DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizercited by 1
  21. 2025MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMscited by 0
  22. 2026Multilingual Fine-Tuning via Localized Gradient Conflict Resolutioncited by 0
  23. 2026Who Bridges Safety? Identifying and Targeting Cross-Lingual Shared Safety Pathwayscited by 0
  24. 2026Training Specialist Models without Reasoning Trajectories for Domain Expert Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages
  2. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
  3. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
  4. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners
  5. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
  6. 2023Letz Translate: Low-Resource Machine Translation for Luxembourgish
  7. 2022Bootstrapping Multilingual Semantic Parsers using Large Language Models
  8. 2024BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment