mlsanityData to 14 Oct 2026

Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

arXiv 2404.04626 ↗First posted 6 Apr 2024cs.CL cs.AIcited by 69

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO), which derives reward signals directly from pairwise preference data, has shown its effectiveness on aligning Large Language Models (LLMs) with human preferences. Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the SFT's effectiveness and its hindrance to the learning capacity towards human-preferred responses, leading to less satisfactory performance. To overcome those limitations, the theoretical understanding of DPO are indispensable but still lacking. To this end, we take a step towards theoretically analyzing and understanding the limitations of DPO. Specifically, we provide an analytical framework using the field theory to analyze the optimization process of DPO. By analyzing the gradient vector field of the DPO loss function, we find that the DPO loss function decreases the probability of producing human dispreferred data at a faster rate than it increases the probability of producing preferred data. This provides theoretical insights for understanding the limitations of DPO discovered in the related research experiments, thereby setting the foundation for its improvement.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 22 references that are in the map. First posted on arXiv 6 Apr 2024.

Builds on

15 of 22 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  2. 2024Direct Preference Optimization with an Offset16 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  7. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  8. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  9. 2017The Numerics of GANscited by 105
  10. 2024Reward Generalization in RLHF: A Topological Perspectivecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 33; 2026: 10.

69 papers.

Sort:

By area:

  1. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  2. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 83
  3. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  4. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  5. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  6. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  7. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 43
  8. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  9. 2025A Survey of Direct Preference Optimizationcited by 36
  10. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  11. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  12. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
38 more
  1. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  2. 2024Bayesian scaling laws for in-context learningcited by 19
  3. 2024SPRec: Self-Play to Debias LLM-based Recommendationcited by 18
  4. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  5. 2024Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimizationcited by 15
  6. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 15
  7. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 13
  8. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  9. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 10
  10. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  11. 2024Length Desensitization in Direct Preference Optimizationcited by 8
  12. 2025RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generationcited by 8
  13. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignmentcited by 7
  14. 2025Enhancing Tool Learning in Large Language Models with Hierarchical Error Checklistscited by 7
  15. 2025SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Modelscited by 7
  16. 2024Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Preference Optimizationcited by 6
  17. 2024GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNetscited by 6
  18. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinementcited by 6
  19. 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
  20. 2024New Desiderata for Direct Preference Optimizationcited by 5
  21. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Modelscited by 5
  22. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  23. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularizationcited by 4
  24. 2024Aligning Visual Contrastive learning models via Preference Optimizationcited by 4
  25. 2024Continual SFT Matches Multimodal RLHF with Negative Supervisioncited by 4
  26. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMscited by 4
  27. 2025CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuningcited by 4
  28. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  29. 2024ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihoodcited by 3
  30. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignmentcited by 3
  31. 2025Empowering LLMs in Task-Oriented Dialogues: A Domain-Independent Multi-Agent Framework and Fine-Tuning Strategycited by 3
  32. 2025SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioningcited by 3
  33. 2025Understand the Implication: Learning to Think for Pragmatic Understandingcited by 3
  34. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlapcited by 3
  35. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  36. 2025Beyond Reward Margin: Rethinking and Resolving Likelihood Displacement in Diffusion Models via Video Generationcited by 3
  37. 2024Learning to Reason via Self-Iterative Process Feedback for Small Language Modelscited by 2
  38. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025What Matters in Data for DPO?
  2. 2025DPO-Shift: Shifting the Distribution of Direct Preference Optimization
  3. 2024Minor DPO reject penalty to increase training robustness
  4. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  5. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  6. 2026LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
  7. 2024Understanding Reference Policies in Direct Preference Optimization
  8. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge