mlsanityData to 14 Oct 2026

Sequence-Level Knowledge Distillation

arXiv 1606.07947 ↗First posted 25 Jun 2016cs.CL cs.LG cs.NEcited by 801

Not on the map. Why?

Cited by 361 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural machine translation (NMT) offers a novel alternative formulation of translation that is potentially simpler than statistical approaches. However to reach competitive performance, NMT models need to be exceedingly large. In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006; Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT. We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model). Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance. It is also significantly better than a baseline model trained without knowledge distillation: by 4.2/1.7 BLEU with greedy decoding/beam search. Applying weight pruning on top of knowledge distillation results in a student model that has 13 times fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.

Key facts. Cited by 801 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 99; 2022: 94; 2023: 98; 2024: 105; 2025: 91; 2026: 313.

801 papers.

Sort:

By area:

  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  5. 2026Reinforcement Learning via Self-Distillationcited by 332
  6. 2022Large Language Models Are Reasoning Teacherscited by 329
  7. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  8. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  10. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  11. 2023A Survey on Model Compression for Large Language Modelscited by 269
  12. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
38 more
  1. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  2. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  3. 2022Continuous diffusion for categorical datacited by 214
  4. 2021Does Knowledge Distillation Really Work?cited by 190
  5. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 187
  6. 2021A Survey on Neural Speech Synthesiscited by 184
  7. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  8. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  9. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  10. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  11. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  12. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  13. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  14. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  15. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 149
  16. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  17. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  18. 2023Getting aligned on representational alignmentcited by 143
  19. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  20. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  21. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  22. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  23. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  24. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 115
  25. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  26. 2021Scaling Laws for Neural Machine Translationcited by 111
  27. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  28. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  29. 2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 101
  30. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrappingcited by 98
  31. 2021Finetuning Pretrained Transformers into RNNscited by 96
  32. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  33. 2021Trustworthy AI: A Computational Perspectivecited by 93
  34. 2021Survey of Low-Resource Machine Translationcited by 92
  35. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 90
  36. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
  37. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 86
  38. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86

50 most cited of 801 All 801 (JSON)