mlsanityData to 7 Oct 2026

DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)

arXiv 2507.07855 ↗First posted 10 Jul 2025cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimization (DPO) cleverly bypasses reward modeling by making an explicit link with a specific normative model of human choice. Our paper elevates this connection to the full generality of DPO's normative framework. Getting there requires reworking human choice theory's textbook path for a better RLHF/ML fit. It elevates the connection to a remarkably broad viewpoint on preference optimization, considering the current panorama of DPO follow-ups. It also unveils unexpected riches for ML, chief among which the support for non-convex losses, the fact that any compliant ML analytical choice can be embedded with any human choice model, and a normative framework's umbrella wide enough to safeguard DPO's extensions (margins, length correction, ...). A toy experiment ``far away'' from the DPO crowd is given.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 68 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

35 of 68 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective20 Feb
  2. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters2 Feb
  3. 2025AlphaPO: Reward Shape Matters for LLM Alignment7 Jan
  4. 2024Weighted-Reward Preference Optimization for Implicit Model Fusion4 Dec
  5. 2024Meta-Learning Objectives for Preference Optimization10 Nov
  6. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization5 Oct
  7. 2024On Extending Direct Preference Optimization to Accommodate Ties25 Sep
  8. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  9. 2024Direct Preference Knowledge Distillation for Large Language Models28 Jun
  10. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence16 Jun
  11. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  12. 2024Self-Improving Robust Preference Optimization3 Jun
7 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep

Other references · 16

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2021Convex optimizationcited by 2,479
  3. 2018Computational Optimal Transportcited by 765
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  7. 2019Learning with Fenchel-Young Lossescited by 117
  8. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  9. 2024Beyond Preferences in AI Alignmentcited by 46
  10. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  11. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  12. 2021Being Properly Impropercited by 11
4 more
  1. 2023Boosting with Tempered Exponential Measurescited by 8
  2. 2020Supervised Learning: No Loss No Crycited by 7
  3. 2025Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Trackcited by 7
  4. 2020All your loss are belong to Bayescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
  2. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  3. 2024New Desiderata for Direct Preference Optimization
  4. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model
  5. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  6. 2025Why DPO is a Misspecified Estimator and How to Fix It
  7. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  8. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications