mlsanityData to 7 Oct 2026

Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling

arXiv 2410.11325 ↗First posted 15 Oct 2024cs.CL cs.AIcited by 85

Not on the map. Why?

Abstract (arXiv)

Recent advances in knowledge distillation (KD) have enabled smaller student models to approach the performance of larger teacher models. However, popular methods such as supervised KD and on-policy KD, are adversely impacted by the knowledge gaps between teacher-student in practical scenarios. Supervised KD suffers from a distribution mismatch between training with a static dataset and inference over final student-generated outputs. Conversely, on-policy KD, which uses student-generated samples for training, can suffer from low-quality training examples with which teacher models are not familiar, resulting in inaccurate teacher feedback. To address these limitations, we introduce Speculative Knowledge Distillation (SKD), a novel approach that leverages cooperation between student and teacher models to generate high-quality training data on-the-fly while aligning with the student's inference-time distribution. In SKD, the student proposes tokens, and the teacher replaces poorly ranked ones based on its own distribution, transferring high-quality knowledge adaptively. We evaluate SKD on various text generation tasks, including translation, summarization, math, and instruction following, and show that SKD consistently outperforms existing KD methods across different domains, data sizes, and model initialization strategies.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

28 of 40 references.

References · 28

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  12. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
16 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2016Sequence-Level Knowledge Distillationcited by 761
  5. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  6. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  7. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  8. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  10. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  11. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  12. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  13. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  14. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  15. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  16. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 64.

85 papers.

Sort:

By area:

  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  2. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  3. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  4. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  6. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  7. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoningcited by 29
  8. 2026Trust Region On-Policy Distillationcited by 26
  9. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  10. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  11. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 22
  12. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 22
38 more
  1. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  2. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  3. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  4. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
  5. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14
  6. 2026Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillationcited by 13
  7. 2026Multi-Turn On-Policy Distillation with Prefix Replaycited by 12
  8. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 12
  9. 2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 10
  10. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 10
  11. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 9
  12. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillationcited by 9
  13. 2026DOPD: Dual On-policy Distillationcited by 9
  14. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  15. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  16. 2026Trust-Region Behavior Blending for On-Policy Distillationcited by 7
  17. 2025Skill-Targeted Adaptive Trainingcited by 6
  18. 2026Self-Distilled Policy Gradientcited by 6
  19. 2025Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillationcited by 5
  20. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
  21. 2026ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillationcited by 5
  22. 2025Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Frameworkcited by 4
  23. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Modelscited by 4
  24. 2025Distillation of Large Language Models via Concrete Score Matchingcited by 4
  25. 2025AdaSwitch: Balancing Exploration and Guidance in Knowledge Distillation via Adaptive Switchingcited by 4
  26. 2026Trace-Based On-Policy Distillation for Masked Diffusion Language Modelscited by 4
  27. 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Modelscited by 4
  28. 2026Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillationcited by 4
  29. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 3
  30. 2025MapCoder-Lite: Distilling Multi-Agent Coding into a Single Small LLMcited by 3
  31. 2026Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memorycited by 3
  32. 2026Tail-Aware Top-k On-Policy Distillationcited by 3
  33. 2025MASTER: Enhancing Large Language Model via Multi-Agent Simulated Teachingcited by 2
  34. 2025AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distributioncited by 2
  35. 2026Reasoning Can Be Restored by Correcting a Few Decision Tokenscited by 2
  36. 2026Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillationcited by 2
  37. 2026β-OPSD: Deriving with Policy Optimization, Training with Self-Distillationcited by 2
  38. 2025SD2: Self-Distilled Sparse Drafterscited by 1

50 most cited of 85 All 85 (JSON)