mlsanityData to 7 Oct 2026

Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation

arXiv 2406.13114 ↗First posted 19 Jun 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have significantly advanced various natural language processing tasks, but deploying them remains computationally expensive. Knowledge distillation (KD) is a promising solution, enabling the transfer of capabilities from larger teacher LLMs to more compact student models. Particularly, sequence-level KD, which distills rationale-based reasoning processes instead of merely final outcomes, shows great potential in enhancing students' reasoning capabilities. However, current methods struggle with sequence level KD under long-tailed data distributions, adversely affecting generalization on sparsely represented domains. We introduce the Multi-Stage Balanced Distillation (BalDistill) framework, which iteratively balances training data within a fixed computational budget. By dynamically selecting representative head domain examples and synthesizing tail domain examples, BalDistill achieves state-of-the-art performance across diverse long-tailed datasets, enhancing both the efficiency and efficacy of the distilled models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 51 of its 80 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

51 of 80 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  9. 2016Sequence-Level Knowledge Distillationcited by 761
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  11. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
38 more
  1. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  2. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  3. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  4. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  5. 2022Large Language Models Are Reasoning Teacherscited by 318
  6. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  7. 2021Parametric Contrastive Learningcited by 217
  8. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  9. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  10. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  11. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  12. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  13. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  14. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  15. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  16. 2021Distilling Virtual Examples for Long-tailed Recognitioncited by 72
  17. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
  18. 2019Sampling Bias in Deep Active Classification: An Empirical Studycited by 36
  19. 2021VideoLT: Large-scale Long-tailed Video Recognitioncited by 35
  20. 2021Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distributioncited by 34
  21. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  22. 2020Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Modelcited by 28
  23. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  24. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  25. 2024Enhancing Text Classification through LLM-Driven Active Learning and Human Annotationcited by 20
  26. 2020Active Sentence Learning by Adversarial Uncertainty Sampling in Discrete Spacecited by 17
  27. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  28. 2023TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewisecited by 15
  29. 2023Pitfalls in Link Prediction with Graph Neural Networks: Understanding the Impact of Target-link Inclusion & Better Practicescited by 13
  30. 2024Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarioscited by 12
  31. 2023Scalable Prompt Generation for Semi-supervised Learning with Language Modelscited by 10
  32. 2023CAT: Causal Audio Transformer for Audio Classificationcited by 10
  33. 2024Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysiscited by 10
  34. 2024The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Educationcited by 9
  35. 2022Making Pretrained Language Models Good Long-tailed Learnerscited by 8
  36. 2022Why only Micro-F1? Class Weighting of Measures for Relation Classificationcited by 8
  37. 2022Benchmarking Long-tail Generalization with Likelihood Splitscited by 8
  38. 2023Long-Tailed Question Answering in an Open Worldcited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  2. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  3. 2025DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Datacited by 6
  4. 2024CSRec: Rethinking Sequential Recommendation from A Causal Perspectivecited by 4
  5. 2025Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  2. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  3. 2026WASD: Wasserstein-based Knowledge Distillation for Large Language Models
  4. 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
  5. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  6. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Models
  7. 2024Mentor-KD: Making Small Language Models Better Multi-step Reasoners
  8. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning