mlsanityData to 14 Oct 2026

A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

arXiv 2410.15595 ↗First posted 21 Oct 2024cs.AI cs.CL cs.LGcited by 27

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical. Direct Preference Optimization (DPO) has emerged as a promising approach for alignment, acting as an RL-free alternative to Reinforcement Learning from Human Feedback (RLHF). Despite DPO's various advancements and inherent limitations, an in-depth review of these aspects is currently lacking in the literature. In this work, we present a comprehensive review of the challenges and opportunities in DPO, covering theoretical analyses, variants, relevant preference datasets, and applications. Specifically, we categorize recent studies on DPO based on key research questions to provide a thorough understanding of DPO's current landscape. Additionally, we propose several future research directions to offer insights on model alignment for the research community. An updated collection of relevant papers can be found on https://github.com/Mr-Loevan/DPO-Survey.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 171 of its 249 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

171 of 249 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  2. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  3. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment2 Feb
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 165

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2023Mistral 7Bcited by 3,164
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2023A Survey of Large Language Modelscited by 2,576
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2024Qwen2 Technical Reportcited by 2,120
  13. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  14. 2019Towards VQA Models That Can Readcited by 1,718
  15. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  16. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  17. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  18. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  19. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  20. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  21. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  22. 2023A Survey on Multimodal Large Language Modelscited by 909
  23. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  24. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  25. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  26. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  27. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  28. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  29. 2024Self-Rewarding Language Modelscited by 681
  30. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  31. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  32. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  33. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  34. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  35. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  36. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  37. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  38. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499

50 most cited of 165 All 165 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 16; 2026: 10.

27 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025A Survey of Direct Preference Optimizationcited by 36
  4. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  5. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  6. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Followingcited by 8
  7. 2026VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoningcited by 8
  8. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  9. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 6
  10. 2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 5
  11. 2025A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solvingcited by 4
  12. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
15 more
  1. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihoodcited by 3
  2. 2025Maximize Your Diffusion: A Study into Reward Maximization and Alignment for Diffusion-based Controlcited by 2
  3. 2025A Systematic Review of Open Datasets Used in Text-to-Image (T2I) Gen AI Model Safetycited by 2
  4. 2025Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixingcited by 2
  5. 2025Lightweight Robust Direct Preference Optimizationcited by 2
  6. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimizationcited by 1
  7. 2026LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Modelscited by 1
  8. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  9. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
  10. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  11. 2026Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Modelscited by 1
  12. 2025GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPOcited by 0
  13. 2025Preference Robustness for DPO with Applications to Public Healthcited by 0
  14. 2025Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMscited by 0
  15. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024New Desiderata for Direct Preference Optimization
  2. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  3. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  4. 2024Minor DPO reject penalty to increase training robustness
  5. 2024Filtered Direct Preference Optimization
  6. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  7. 2024Direct Preference Optimization with an Offset
  8. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms