mlsanityData to 7 Oct 2026

Robust Preference Optimization through Reward Model Distillation

arXiv 2405.19316 ↗First posted 29 May 2024cs.LG cs.CLcited by 67

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Language model (LM) post-training (or alignment) involves maximizing a reward function that is derived from preference annotations. Direct Preference Optimization (DPO) is a popular offline alignment method that trains a policy directly on preference data without the need to train a reward model or apply reinforcement learning. However, the empirical evidence suggests that DPO typically assigns implicit rewards that overfit, and trend towards infinite magnitude. This frequently leads to degenerate policies, sometimes causing even the probabilities of the preferred generations to go to zero. In this work, we analyze this phenomenon and use distillation to get a better proxy for the true preference distribution over generation pairs: we train the LM such that its induced implicit reward, i.e., the scaled log-likelihood ratio of the model to the reference model, matches an explicit reward model trained on the preference data. Moreover, to account for uncertainty in the reward model we are distilling from, we optimize against a family of reward models that, as a whole, is likely to include at least one reasonable proxy for the preference distribution. Our results show that distilling from such a family of reward models leads to improved robustness to distribution shift in preference annotations, while preserving the simple supervised nature of DPO.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 58 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

49 of 58 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  4. 2024Token-level Direct Preference Optimization18 Apr
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  7. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  8. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  9. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  10. 2024Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration25 Feb
  11. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  12. 2024Direct Preference Optimization with an Offset16 Feb
3 more
  1. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  2. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  10. 2018Born Again Neural Networkscited by 539
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
22 more
  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  6. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  7. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  8. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  9. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  10. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  11. 2023Controlled Decoding from Language Modelscited by 133
  12. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  13. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  14. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  15. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  16. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  17. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  18. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  19. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  20. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  21. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  22. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 38; 2026: 16.

67 papers.

Sort:

By area:

  1. 2024Nemotron-4 340B Technical Reportcited by 143
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  3. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  4. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  5. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  6. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  7. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  8. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  9. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  10. 2025A Survey of Direct Preference Optimizationcited by 35
  11. 2024InfAlign: Inference-aware language model alignmentcited by 34
  12. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Levelcited by 22
38 more
  1. 2025RobustFlow: Towards Robust Agentic Workflow Generationcited by 19
  2. 2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14
  3. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  4. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  5. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignmentcited by 10
  6. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  7. 2025Frictional Agent Alignment Framework: Slow Down and Don't Break Thingscited by 10
  8. 2025Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Samplingcited by 9
  9. 2024PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Casescited by 8
  10. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  11. 2025Mitigating Preference Hacking in Policy Optimization with Pessimismcited by 7
  12. 2025Multiplayer Nash Preference Optimizationcited by 7
  13. 2024Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimizationcited by 6
  14. 2025Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomescited by 6
  15. 2024Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Bothcited by 5
  16. 2025Iterative Label Refinement Matters More than Preference Optimization under Weak Supervisioncited by 5
  17. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  18. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  19. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  20. 2025Design Considerations in Offline Preference-based RLcited by 4
  21. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 4
  22. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferencescited by 4
  23. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  24. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimizationcited by 3
  25. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlapcited by 3
  26. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  27. 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3
  28. 2026Reward Models Inherit Value Biases from Pretrainingcited by 3
  29. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
  30. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
  31. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  32. 2025Adaptive Preference Optimization with Uncertainty-aware Utility Anchorcited by 2
  33. 2025Reward Models are Metrics in a Trench Coatcited by 2
  34. 2025Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policiescited by 2
  35. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  36. 2026LibraGen: Playing a Balance Game in Subject-Driven Video Generationcited by 2
  37. 2025Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Samplingcited by 1
  38. 2025V.I.P. : Iterative Online Preference Distillation for Efficient Video Diffusion Modelscited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
  2. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  4. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  5. 2024Bootstrapping Language Models with DPO Implicit Rewards
  6. 2025Explicit Preference Optimization: No Need for an Implicit Reward Model
  7. 2025Why DPO is a Misspecified Estimator and How to Fix It
  8. 2024New Desiderata for Direct Preference Optimization