mlsanityData to 14 Oct 2026

KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation

arXiv 2504.19024 ↗First posted 26 Apr 2025cs.CLcited by 3

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

We propose a novel k-step return estimation method (called KETCHUP) for Reinforcement Learning(RL)-based knowledge distillation (KD) in text generation tasks. Our idea is to induce a K-step return by using the Bellman Optimality Equation for multiple steps. Theoretical analysis shows that this K-step formulation reduces the variance of the gradient estimates, thus leading to improved RL optimization especially when the student model size is large. Empirical evaluation on three text generation tasks demonstrates that our approach yields superior performance in both standard task metrics and large language model (LLM)-based evaluation. These results suggest that our K-step return induction offers a promising direction for enhancing RL-based KD in LLM research.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 49 of its 84 references that are in the map. First posted on arXiv 26 Apr 2025.

Builds on

49 of 84 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Best Practices for Distilling Large Language Models into BERT for Web Search Ranking7 Nov
  2. 2024LLMR: Knowledge Distillation with a Large Language Model-Induced Reward19 Sep
  3. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning20 Feb
  4. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillation27 Jul
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes23 Jun
  6. 2023Accurate Knowledge Distillation with n-best Reranking20 May

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2015Trust Region Policy Optimizationcited by 2,580
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
31 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  8. 2016Sequence-Level Knowledge Distillationcited by 801
  9. 2017Deep Reinforcement Learning that Matterscited by 754
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  12. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  13. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  14. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  15. 2024LoRA Learns Less and Forgets Lesscited by 328
  16. 2022Teaching Small Language Models to Reasoncited by 284
  17. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  18. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  19. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  20. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  21. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  22. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  23. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  24. 2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
  25. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 28
  26. 2020Reanalysis of Variance Reduced Temporal Difference Learningcited by 23
  27. 2017Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselinescited by 20
  28. 2018Contextual Neural Model for Translating Bilingual Multi-Speaker Conversationscited by 15
  29. 2018TD-Regularized Actor-Critic Methodscited by 14
  30. 2017Why Do Neural Dialog Systems Generate Short and Meaningless Replies? A Comparison between Dialog and Translationcited by 12
  31. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  2. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 6
  3. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation
  2. 2025Knowledge Distillation with Training Wheels
  3. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning
  4. 2026Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
  5. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models
  6. 2026X-KD: General Experiential Knowledge Distillation for Large Language Models
  7. 2024Direct Preference Knowledge Distillation for Large Language Models
  8. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models