mlsanityData to 14 Oct 2026

f-Divergence Minimization for Sequence-Level Knowledge Distillation

arXiv 2307.15190 ↗First posted 27 Jul 2023cs.CL cs.LGcited by 114

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation (KD) is the process of transferring knowledge from a large model to a small one. It has gained increasing attention in the natural language processing community, driven by the demands of compressing ever-growing language models. In this work, we propose an f-DISTILL framework, which formulates sequence-level knowledge distillation as minimizing a generalized f-divergence function. We propose four distilling variants under our framework and show that existing SeqKD and ENGINE approaches are approximations of our f-DISTILL methods. We further derive step-wise decomposition for our f-DISTILL, reducing intractable sequence-level divergence to word-level losses that can be computed in a tractable manner. Experiments across four datasets show that our methods outperform existing KD approaches, and that our symmetric distilling losses can better force the student to learn from the teacher distribution.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 48 of its 65 references that are in the map. First posted on arXiv 27 Jul 2023.

Builds on

48 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers2 Jun

Other references · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
35 more
  1. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  3. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  4. 2016Sequence-Level Knowledge Distillationcited by 801
  5. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  6. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  7. 2017Non-Autoregressive Neural Machine Translationcited by 463
  8. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  9. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  10. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  11. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  12. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  13. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  14. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  15. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  16. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 145
  17. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
  18. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 74
  19. 2020Pre-trained Summarization Distillationcited by 71
  20. 2021Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularizationcited by 62
  21. 2019Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering Systemcited by 43
  22. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  23. 2021Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?cited by 38
  24. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
  25. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Datacited by 32
  26. 2021Commonsense-Focused Dialogues for Response Generation: An Empirical Studycited by 30
  27. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27
  28. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
  29. 2021Layer-wise Model Pruning based on Mutual Informationcited by 19
  30. 2017Why Do Neural Dialog Systems Generate Short and Meaningless Replies? A Comparison between Dialog and Translationcited by 12
  31. 2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
  32. 2020Improving Task-Agnostic BERT Distillation with Layer Mapping Searchcited by 8
  33. 2022What Makes Data-to-Text Generation Hard for Pretrained Language Models?cited by 8
  34. 2018Efficient Contextualized Representation: Language Model Pruning for Sequence Labelingcited by 6
  35. 2020Bridging Maximum Likelihood and Adversarial Learning via α-Divergencecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 31; 2025: 31; 2026: 47.

114 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  3. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  4. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  5. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  6. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 84
  7. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  8. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 78
  9. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  10. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  11. 2026Self-Distilled Agentic Reinforcement Learningcited by 55
  12. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 53
38 more
  1. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  2. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 42
  3. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 40
  4. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 36
  5. 2024A Survey of Small Language Modelscited by 31
  6. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 30
  7. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 29
  8. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 26
  9. 2026On the Position Bias of On-Policy Distillationcited by 26
  10. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  11. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  12. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 23
  13. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMscited by 22
  14. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
  15. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 22
  16. 2024Direct Preference Knowledge Distillation for Large Language Modelscited by 21
  17. 2023Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Expertscited by 17
  18. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillationcited by 17
  19. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 16
  20. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
  21. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  22. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  23. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 12
  24. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairscited by 11
  25. 2024Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignmentcited by 11
  26. 2024Efficient Inference for Large Language Model-based Generative Recommendationcited by 11
  27. 2026OVD: On-policy Verbal Distillationcited by 11
  28. 2025Training Domain Draft Models for Speculative Decoding: Best Practices and Insightscited by 10
  29. 2025DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Modelscited by 10
  30. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 10
  31. 2024SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Modelscited by 9
  32. 2025Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanationcited by 9
  33. 2026Hybrid Policy Distillation for LLMscited by 9
  34. 2025A Dual-Space Framework for General Knowledge Distillation of Large Language Modelscited by 8
  35. 2025AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoderscited by 8
  36. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 8
  37. 2026Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalizationcited by 8
  38. 2024NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Modelscited by 7

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ToDi: Token-wise Distillation via Fine-Grained Divergence Control
  2. 2026WASD: Wasserstein-based Knowledge Distillation for Large Language Models
  3. 2023MiniLLM: On-Policy Distillation of Large Language Models
  4. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
  5. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  6. 2022Knowledge Distillation of Russian Language Models with Reduction of Vocabulary
  7. 2025Delta Knowledge Distillation for Large Language Models
  8. 2025Vocabulary Expansion of Large Language Models via Kullback-Leibler-Based Self-Distillation