mlsanityData to 14 Oct 2026

BPO: Revisiting Preference Modeling in Direct Preference Optimization

arXiv 2506.03557 ↗First posted 4 Jun 2025cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) have emerged as a popular method for aligning Large Language Models (LLMs) with human preferences. While DPO effectively preserves the relative ordering between chosen and rejected responses through pairwise ranking losses, it often neglects absolute reward magnitudes. This oversight can decrease the likelihood of chosen responses and increase the risk of generating out-of-distribution responses, leading to poor performance. We term this issue Degraded Chosen Responses (DCR).To address this issue, we propose Balanced Preference Optimization (BPO), a novel framework that dynamically balances the optimization of chosen and rejected responses through two key components: balanced reward margin and gap adaptor. Unlike previous methods, BPO can fundamentally resolve DPO's DCR issue, without introducing additional constraints to the loss function. Experimental results on multiple mathematical reasoning tasks show that BPO significantly outperforms DPO, improving accuracy by +10.1% with Llama-3.1-8B-Instruct (18.8% to 28.9%) and +11.7% with Qwen2.5-Math-7B (35.0% to 46.7%). It also surpasses DPO variants by +3.6% over IPO (43.1%), +5.0% over SLiC (41.7%), and +3.1% over Cal-DPO (43.6%) on the same model. Remarkably, our algorithm requires only a single line of code modification, making it simple to implement and fully compatible with existing DPO-based frameworks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 29 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

25 of 29 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  2. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  3. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment28 May
  4. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks23 Apr
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  11. 2023Mitigating the Alignment Tax of RLHF12 Sep
  12. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
1 more
  1. 2023Nash Learning from Human Feedbackcited by 224

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference Optimization by Estimating the Ratio of the Data Distribution
  2. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
  3. 2026HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
  4. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
  5. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
  6. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  7. 2025A Survey of Direct Preference Optimization
  8. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective