mlsanityData to 14 Oct 2026

Why DPO is a Misspecified Estimator and How to Fix It

arXiv 2510.20413 ↗First posted 23 Oct 2025cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct alignment algorithms such as Direct Preference Optimization (DPO) fine-tune models based on preference data, using only supervised learning instead of two-stage reinforcement learning with human feedback (RLHF). We show that DPO encodes a statistical estimation problem over reward functions induced by a parametric policy class. When the true reward function that generates preferences cannot be realized via the policy class, DPO becomes misspecified, resulting in failure modes such as preference order reversal, worsening of policy reward, and high sensitivity to the input preference data distribution. On the other hand, we study the local behavior of two-stage RLHF for a parametric class and relate it to a natural gradient step in policy space. Our fine-grained geometric characterization allows us to propose AuxDPO, which introduces additional auxiliary variables in the DPO loss function to help move towards the RLHF solution in a principled manner and mitigate the misspecification in DPO. We empirically demonstrate the superior performance of AuxDPO on didactic bandit settings as well as LLM alignment tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 26 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

19 of 26 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift10 Jul
  2. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  3. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 11

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  7. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  8. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  9. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  10. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  11. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  2. 2026Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterizationcited by 1
  3. 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  2. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
  3. 2024Minor DPO reject penalty to increase training robustness
  4. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimization
  5. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss
  6. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  7. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  8. 2025C2-DPO: Constrained Controlled Direct Preference Optimization