mlsanityData to 14 Oct 2026

A Survey of Direct Preference Optimization

arXiv 2503.11701 ↗First posted 12 Mar 2025cs.LGcited by 36

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated unprecedented generative capabilities, yet their alignment with human values remains critical for ensuring helpful and harmless deployments. While Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful paradigm for aligning LLMs with human preferences, its reliance on complex reward modeling introduces inherent trade-offs in computational efficiency and training stability. In this context, Direct Preference Optimization (DPO) has recently gained prominence as a streamlined alternative that directly optimizes LLMs using human preferences, thereby circumventing the need for explicit reward modeling. Owing to its theoretical elegance and computational efficiency, DPO has rapidly attracted substantial research efforts exploring its various implementations and applications. However, this field currently lacks systematic organization and comparative analysis. In this survey, we conduct a comprehensive overview of DPO and introduce a novel taxonomy, categorizing previous works into four key dimensions: data strategy, learning framework, constraint mechanism, and model property. We further present a rigorous empirical analysis of DPO variants across standardized benchmarks. Additionally, we discuss real-world applications, open challenges, and future directions for DPO. This work delivers both a conceptual framework for understanding DPO and practical guidance for practitioners, aiming to advance robust and generalizable alignment paradigms. All collected resources are available and will be continuously updated at https://github.com/liushunyu/awesome-direct-preference-optimization.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 236 of its 266 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

236 of 266 references.

Closest ancestors · 122

Same direction, newest first

  1. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective20 Feb
  2. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization19 Feb
  3. 2025Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Data8 Feb
  4. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters2 Feb
  5. 2025Online Preference Alignment for Language Models via Count-based Exploration22 Jan
  6. 2025DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization5 Jan
  7. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agents3 Jan
  8. 2024No Preference Left Behind: Group Distributional Preference Optimization28 Dec
  9. 2024Understanding the Logic of Direct Preference Alignment through Logic23 Dec
  10. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  11. 2024Constrain Alignment with Sparse Autoencoders12 Nov
  12. 2024IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization9 Nov
38 more
  1. 2024TODO: Enhancing LLM Alignment with Ternary Preferences2 Nov
  2. 2024Aligning CodeLLMs with Direct Preference Optimization24 Oct
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications21 Oct
  4. 2024Holistic Utility Preference Learning for Listwise Alignment17 Oct
  5. 2024A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement17 Oct
  6. 2024Preference Optimization with Multi-Sample Comparisons16 Oct
  7. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithms15 Oct
  8. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization14 Oct
  9. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins12 Oct
  10. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  11. 2024Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both11 Oct
  12. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  13. 2024Accelerated Preference Optimization for Large Language Model Alignment8 Oct
  14. 2024SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks7 Oct
  15. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights6 Oct
  16. 2024Permutative Preference Alignment from Listwise Ranking of Human Judgments6 Oct
  17. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization5 Oct
  18. 2024FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization1 Oct
  19. 2024The Crucial Role of Samplers in Online Direct Preference Optimization29 Sep
  20. 2024On Extending Direct Preference Optimization to Accommodate Ties25 Sep
  21. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey17 Sep
  22. 2024AIPO: Improving Training Objective for Iterative Preference Optimization13 Sep
  23. 2024Geometric-Averaged Preference Optimization for Soft Preference Labels10 Sep
  24. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization5 Sep
  25. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey4 Sep
  26. 2024Selective Preference Optimization via Token-Level Reward Function Estimation24 Aug
  27. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization14 Aug
  28. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  29. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift26 Jul
  30. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment19 Jul
  31. 2024Understanding Reference Policies in Direct Preference Optimization18 Jul
  32. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  33. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  34. 2024Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment3 Jul
  35. 2024PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning25 Jun
  36. 2024Direct Multi-Turn Preference Optimization for Language Agents21 Jun
  37. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence16 Jun
  38. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun

50 newest of 122 All 122 (JSON)

Other references · 114

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
38 more
  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2023A Survey on Multimodal Large Language Modelscited by 909
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  13. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  14. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  15. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  16. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  17. 2024Self-Rewarding Language Modelscited by 681
  18. 2023A Comprehensive Overview of Large Language Modelscited by 648
  19. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  20. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  21. 2024Large Language Models: A Surveycited by 525
  22. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  23. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  24. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  25. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  26. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  27. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  28. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  29. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  30. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  31. 2023AI Alignment: A Comprehensive Surveycited by 263
  32. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  33. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 252
  34. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  35. 2024Iterative Reasoning Preference Optimizationcited by 246
  36. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  37. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  38. 2023Nash Learning from Human Feedbackcited by 224

50 most cited of 114 All 114 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 15.

36 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  3. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  4. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  5. 2025Learning Dynamics of VLM Finetuningcited by 17
  6. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  7. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Modelscited by 7
  8. 2025Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedbackcited by 6
  9. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 6
  10. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  11. 2025Supervised Optimism Correction: Be Confident When LLMs Are Surecited by 5
  12. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
24 more
  1. 2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 5
  2. 2025Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compressioncited by 4
  3. 2025Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMscited by 4
  4. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  5. 2025Reverse Preference Optimization for Complex Instruction Followingcited by 3
  6. 2025Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generationcited by 2
  7. 2025Tree of Preferences for Diversified Recommendationcited by 2
  8. 2026Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulationcited by 2
  9. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimizationcited by 1
  10. 2026SPA-Cache: Singular Proxies for Adaptive Caching in Diffusion Language Modelscited by 1
  11. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  12. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  13. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigmcited by 0
  14. 2025Direct Preference Optimization for Adaptive Concept-based Explanationscited by 0
  15. 2025Intra-Trajectory Consistency for Reward Modelingcited by 0
  16. 2025A Novel Self-Evolution Framework for Large Language Modelscited by 0
  17. 2025When Are Two RLHF Objectives the Same?cited by 0
  18. 2026Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learningcited by 0
  19. 2026On Reasoning Behind Next Occupation Recommendationcited by 0
  20. 2026TokenRatio: Principled Token-Level Preference Optimization via Ratio Matchingcited by 0
  21. 2026Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecscited by 0
  22. 2026Logic-VLA: A Temporal Logic Conditioned Vision-Language-Action Modelcited by 0
  23. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignmentcited by 0
  24. 2026Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024New Desiderata for Direct Preference Optimization
  2. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  3. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  4. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  5. 2024Multi-Reference Preference Optimization for Large Language Models
  6. 2024Active Preference Learning for Large Language Models
  7. 2024Minor DPO reject penalty to increase training robustness
  8. 2025What Matters in Data for DPO?