mlsanityData to 14 Oct 2026

C2-DPO: Constrained Controlled Direct Preference Optimization

arXiv 2502.17507 ↗First posted 22 Feb 2025cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct preference optimization (textttDPO) has emerged as a promising approach for solving the alignment problem in AI. In this paper, we make two counter-intuitive observations about textttDPO. First, we show that textttDPO loss could be derived by starting from an alternative optimization problem that only defines the KL guardrail on in-sample responses, unlike the original RLHF problem where guardrails are defined on the entire distribution. Second, we prove a surprising property of this alternative optimization problem, namely that under its optimal policy, both preferred and rejected responses tend to decrease in probability, a phenomenon typically displayed by DPO in practice. To control this behavior, we propose a set of constraints designed to limit the displacement of probability mass between the preferred and rejected responses in the reference and target policies. The resulting algorithm, which we call Constrained Controlled DPO (textttC2-DPO), has a meaningful RLHF interpretation. By hedging against the displacement, textttC2-DPO provides practical improvements over vanilla textttDPO when aligning several language models using standard preference datasets.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 45 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

33 of 45 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective20 Feb
  2. 2025DPO-Shift: Shifting the Distribution of Direct Preference Optimization11 Feb
  3. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms5 Feb
  4. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  5. 2024TODO: Enhancing LLM Alignment with Ternary Preferences2 Nov
  6. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimization29 Oct
  7. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  8. 2024AIPO: Improving Training Objective for Iterative Preference Optimization13 Sep
  9. 2024Minor DPO reject penalty to increase training robustness19 Aug
  10. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  11. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  12. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward11 Jun
8 more
  1. 2024Robust Preference Optimization through Reward Model Distillation29 May
  2. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributions30 Apr
  3. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  4. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective6 Apr
  5. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  6. 2024Direct Preference Optimization with an Offset16 Feb
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 13

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2019When Does Label Smoothing Help?cited by 988
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 20242 OLMo 2 Furiouscited by 340
  8. 2024Iterative Reasoning Preference Optimizationcited by 246
  9. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  10. 2024Nemotron-4 340B Technical Reportcited by 144
  11. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  12. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
1 more
  1. 2022Efficient computation of rankings from pairwise comparisonscited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
  2. 2026Displacement-Resistant Extensions of DPO with Nonconvex f-Divergencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  2. 2026LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
  3. 2025Why DPO is a Misspecified Estimator and How to Fix It
  4. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  5. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  6. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
  7. 2026CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints