mlsanityData to 14 Oct 2026

Rethinking DPO: The Role of Rejected Responses in Preference Misalignment

arXiv 2506.12725 ↗First posted 15 Jun 2025cs.AI cs.CLcited by 15

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives -- increasing the generation probability of chosen responses while reducing that of rejected responses -- due to the dominant influence of rejected responses on the loss function. This imbalance leads to suboptimal performance in promoting preferred responses. In this work, we systematically analyze the limitations of DPO and existing algorithms designed to achieve the objectives stated above. To address these limitations, we propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO. Through theoretical analysis and empirical evaluations, we demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 33 references that are in the map. First posted on arXiv 15 Jun 2025.

Builds on

19 of 33 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Minor DPO reject penalty to increase training robustness19 Aug
  2. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  3. 2024Preference Learning Algorithms Do Not Learn Preference Rankings29 May
  4. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  5. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  6. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 9

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  7. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  9. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2026Colosseum: Auditing Collusion in Cooperative Multi-Agent Systemscited by 18
  2. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Modelscited by 7
  3. 2026Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectoriescited by 4
  4. 2025Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shiftcited by 2
  5. 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shiftcited by 2
  6. 2025Learning from Self Critique and Refinement for Faithful LLM Summarizationcited by 1
  7. 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
  8. 2026Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?cited by 1
  9. 2026Gradient-Gated DPO: Stabilizing Preference Optimization in Language Modelscited by 1
  10. 2026A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationalescited by 1
  11. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  12. 2026Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMscited by 0
3 more
  1. 2026KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignmentcited by 0
  2. 2026PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviorscited by 0
  3. 2026TrustMI: Causally controlling how assistants trust their userscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains
  2. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
  3. 2025What Matters in Data for DPO?
  4. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization
  5. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
  6. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimization
  7. 2025Why DPO is a Misspecified Estimator and How to Fix It
  8. 2025DPO-Shift: Shifting the Distribution of Direct Preference Optimization