mlsanityData to 7 Oct 2026

Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner

arXiv 2604.18239 ↗First posted 20 Apr 2026cs.LG cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization is widely used to align large language models (LLMs) with human preferences. However, many margin-based methods also suppress the chosen response when they try to suppress the rejected one, and there is no general way to prevent this across different objectives. We address this issue with a unified incentive-score decomposition of preference optimization, revealing that different objectives share the same local update directions and differ only in their scalar weights. This decomposition provides a common framework for analyzing objectives that were previously studied in separate settings. Building on this decomposition, by analyzing the dynamics of the chosen/rejected likelihoods, we identify the disentanglement band (DB), a simple, testable condition that tells us when training can follow the desired path: suppress the loser while preserving the winner, possibly after an early stage. Using the DB, we propose reward calibration (RC), a plug-and-play method that adaptively rebalances the updates for chosen and rejected responses to satisfy the DB, without redesigning the base objective. Empirical results show that RC leads to more disentangled dynamics, with better downstream performance observed across several settings. Our code is available at https://github.com/IceyWuu/DisentangledPreferenceOptimization.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 58 of its 78 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

58 of 78 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Token-Importance Guided Direct Preference Optimization26 May
  2. 2025Preference Optimization by Estimating the Ratio of the Data Distribution26 May
  3. 2025Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models12 May
  4. 2025On a Connection Between Imitation Learning and RLHF7 Mar
  5. 2024A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement17 Oct
  6. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  7. 2024Learning Dynamics of LLM Finetuning15 Jul
  8. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward11 Jun
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  10. 2024Token-level Direct Preference Optimization18 Apr
  11. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  12. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
7 more
  1. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  2. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  3. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
27 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  6. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  7. 2023Nash Learning from Human Feedbackcited by 220
  8. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  9. 2020Telescoping Density-Ratio Estimationcited by 108
  10. 2021Density Ratio Estimation via Infinitesimal Classificationcited by 59
  11. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  12. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 30
  13. 2024Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inferencecited by 16
  14. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  15. 2017Trimmed Density Ratio Estimationcited by 15
  16. 2025Density Ratio Estimation with Conditional Probability Pathscited by 13
  17. 2025Dequantified Diffusion-Schr"odinger Bridge for Density Ratio Estimationcited by 11
  18. 2023SciRE-Solver: Accelerating Diffusion Models Sampling by Score-integrand Solver with Recursive Differencecited by 10
  19. 2025EVODiff: Entropy-aware Variance Optimized Diffusion Inferencecited by 10
  20. 2023Sparse Variational Student-t Processescited by 8
  21. 2023Bayesian Gaussian Process ODEs via Double Normalizing Flowscited by 7
  22. 2023Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processescited by 6
  23. 2025Enhancing Adversarial Robustness with Conformal Prediction: A Framework for Guaranteed Model Reliabilitycited by 6
  24. 2024Entropy-Informed Weighting Channel Normalizing Flow for Deep Generative Modelscited by 5
  25. 2026Large Language Model as Token Compressor and Decompressorcited by 4
  26. 2022Tighter Bound Estimation for Efficient Biquadratic Optimization Over Unit Spherescited by 3
  27. 2026One-Step Score-Based Density Ratio Estimationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Smaller Models, Better Rejects: Preference Distillation Scalingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
  2. 2026Disentangling Optimization Scale from Preference Scale in DPO
  3. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
  4. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  5. 2026SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization
  6. 2026REAR: Test-time Preference Realignment through Reward Decomposition
  7. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimization
  8. 2026Masking Frequent Tokens Sharpens Direct Preference Optimization