mlsanityData to 7 Oct 2026

Direct Preference Knowledge Distillation for Large Language Models

arXiv 2406.19774 ↗First posted 28 Jun 2024cs.CLcited by 19

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

In the field of large language models (LLMs), Knowledge Distillation (KD) is a critical technique for transferring capabilities from teacher models to student models. However, existing KD methods face limitations and challenges in distillation of LLMs, including efficiency and insufficient measurement capabilities of traditional KL divergence. It is shown that LLMs can serve as an implicit reward function, which we define as a supplement to KL divergence. In this work, we propose Direct Preference Knowledge Distillation (DPKD) for LLMs. DPKD utilizes distribution divergence to represent the preference loss and implicit reward function. We re-formulate KD of LLMs into two stages: first optimizing and objective consisting of implicit reward and reverse KL divergence and then improving the preference probability of teacher outputs over student outputs. We conducted experiments and analysis on various datasets with LLM parameters ranging from 120M to 13B and demonstrate the broad applicability and effectiveness of our DPKD approach. Meanwhile, we prove the value and effectiveness of the introduced implicit reward and output preference in KD through experiments and theoretical analysis. The DPKD method outperforms the baseline method in both output response precision and exact match percentage. Code and data are available at https://aka.ms/dpkd.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 47 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

29 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  3. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 22

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Knowledge Distillation: A Surveycited by 1,578
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2016Sequence-Level Knowledge Distillationcited by 761
10 more
  1. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  2. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  3. 2019On the Efficacy of Knowledge Distillationcited by 377
  4. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  5. 2022Knowledge Distillation from A Stronger Teachercited by 204
  6. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  7. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  8. 2020LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuningcited by 25
  9. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 24
  10. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 5.

19 papers.

Sort:

By area:

  1. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  2. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  3. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  4. 2025MemLoRA: Distilling Expert Adapters for On-Device Memory Systemscited by 7
  5. 2026SODA: Semi On-Policy Black-Box Distillation for Large Language Modelscited by 6
  6. 2024Understanding the Logic of Direct Preference Alignment through Logiccited by 5
  7. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 4
  8. 2025Distillation of Large Language Models via Concrete Score Matchingcited by 4
  9. 2025From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMscited by 1
  10. 2025Relative Overfitting and Accept-Reject Frameworkcited by 1
  11. 2025Online Knowledge Distillation with Reward Guidancecited by 0
  12. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimizationcited by 0
7 more
  1. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0
  2. 2025Preference Distillation via Value based Reinforcement Learningcited by 0
  3. 2025Is Implicit Knowledge Enough for LLMs? A RAG Approach for Tree-based Structurescited by 0
  4. 2026STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Modelscited by 0
  5. 2026ExecTune: Effective Steering of Black-Box LLMs with Guide Modelscited by 0
  6. 2026When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillationcited by 0
  7. 2026Smaller Models, Better Rejects: Preference Distillation Scalingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs
  2. 2026WASD: Wasserstein-based Knowledge Distillation for Large Language Models
  3. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  4. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  5. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
  6. 2026Hybrid Policy Distillation for LLMs
  7. 2025Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models
  8. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions