mlsanityData to 7 Oct 2026

Displacement-Resistant Extensions of DPO with Nonconvex f-Divergences

arXiv 2602.06788 ↗First posted 6 Feb 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

DPO and related algorithms align language models by directly optimizing the RLHF objective: find a policy that maximizes the Bradley-Terry reward while staying close to a reference policy through a KL divergence penalty. Previous work showed that this approach could be further generalized: the original problem remains tractable even if the KL divergence is replaced by a family of f-divergence with a convex generating function f. Our first contribution is to show that convexity of f is not essential. Instead, we identify a more general condition, referred to as DPO-inducing, that precisely characterizes when the RLHF problem remains tractable. Our next contribution is to establish a second condition on f that is necessary to prevent probability displacement, a known empirical phenomenon in which the probabilities of the winner and the loser responses approach zero. We refer to any f that satisfies this condition as displacement-resistant. We finally focus on a specific DPO-inducing and displacement-resistant f, leading to our novel SquaredPO loss. Compared to DPO, this new loss offers stronger theoretical guarantees while performing competitively in practice.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 35 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

22 of 35 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025C2-DPO: Constrained Controlled Direct Preference Optimization22 Feb
  2. 2025KL Penalty Control via Perturbation for Direct Preference Optimization18 Feb
  3. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  4. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimization29 Oct
  5. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  6. 2024AIPO: Improving Training Objective for Iterative Preference Optimization13 Sep
  7. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  8. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  9. 2024Discovering Preference Optimization Algorithms with and for Large Language Models12 Jun
  10. 2024Robust Preference Optimization through Reward Model Distillation29 May
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  12. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
3 more
  1. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  2. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  3. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb

Other references · 7

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  7. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
  2. 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignment
  3. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
  4. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
  5. 2026Disentangling Optimization Scale from Preference Scale in DPO
  6. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  7. 2024Entropy Controllable Direct Preference Optimization
  8. 2026Rethinking the Divergence Regularization in LLM RL