mlsanityData to 14 Oct 2026

Gradient Imbalance in Direct Preference Optimization

arXiv 2502.20847 ↗First posted 28 Feb 2025cs.LGcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has been proposed as a promising alternative to Proximal Policy Optimization (PPO) based Reinforcement Learning with Human Feedback (RLHF). However, empirical evaluations consistently reveal suboptimal performance in DPO compared to common RLHF pipelines. In this work, we conduct a systematic analysis of DPO's training dynamics and identify gradient imbalance as a critical limitation. We demonstrate theoretically and empirically that this imbalance perturbs optimization trajectories, destabilizes learning, and induces suboptimal convergence. To address this issue, we propose Balanced-DPO, a simple yet effective modification to the DPO objective that introduces a computationally efficient gradient reweighting mechanism. Our experiments demonstrate the effectiveness of Balanced-DPO, validating the theoretical findings and confirming that addressing gradient imbalance is key to improving DPO's performance, highlighting a promising direction for future research.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 22 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

18 of 22 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization14 Oct
  2. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  3. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks23 Apr
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  9. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  10. 2023Do Language Models Know When They're Hallucinating References?cited by 116

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
  2. 2026Where does output diversity collapse in post-training?cited by 11
  3. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updatescited by 1
  4. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  5. 2026Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shiftcited by 0
  6. 2026Disentangling Optimization Scale from Preference Scale in DPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  2. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  3. 2025Why DPO is a Misspecified Estimator and How to Fix It
  4. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  5. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
  6. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss
  7. 2024Minor DPO reject penalty to increase training robustness
  8. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs