mlsanityData to 14 Oct 2026

Online Knowledge Distillation with Reward Guidance

arXiv 2505.18952 ↗First posted 25 May 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

This work studies knowledge distillation (KD) for large language models (LLMs) through preference optimization. We propose a reward-guided imitation learning framework for sequential KD, formulating a min-max optimization problem between the policy and reward model (RM) to minimize the performance gap between the student and teacher policies. Specifically, the reward optimization is constrained to achieve near-optimality within a confidence set for preference alignment. For preference data construction, we explore both offline and online preference-based KD. Additionally, we reformulate the RM using the Q-value function and extend the framework to white-box KD, where the teacher policy's predicted probabilities are accessible. Theoretical analysis and empirical results demonstrate the effectiveness of the proposed framework.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 59 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

45 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Direct Preference Knowledge Distillation for Large Language Models28 Jun
  2. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2024Qwen2 Technical Reportcited by 2,120
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
29 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  6. 2016Sequence-Level Knowledge Distillationcited by 801
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  9. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  11. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  12. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  13. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  14. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  15. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  16. 2024Iterative Reasoning Preference Optimizationcited by 246
  17. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  18. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  19. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  20. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  21. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 74
  22. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  23. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  24. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  25. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  26. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 43
  27. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairscited by 11
  28. 2022Explicit Knowledge Transfer for Weakly-Supervised Code Generationcited by 4
  29. 2024Adversarial Moment-Matching Distillation of Large Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026X-KD: General Experiential Knowledge Distillation for Large Language Models
  2. 2025Preference Distillation via Value based Reinforcement Learning
  3. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Models
  4. 2025Knowledge Distillation with Training Wheels
  5. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoning
  6. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning
  7. 2025Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
  8. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models