mlsanityData to 7 Oct 2026

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

arXiv 2305.18290 ↗First posted 29 May 2023cs.LG cs.AI cs.CLcited by 7,916

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

Key facts. Cited by 7,916 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 60 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

33 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
19 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  7. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  8. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  9. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  10. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  11. 2015Contextual Dueling Banditscited by 101
  12. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  13. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  14. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  15. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  16. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  17. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  18. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  19. 2022Human Preferences as Dueling Banditscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 223; 2024: 1851; 2025: 3090; 2026: 2752.

7,916 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2024Mixtral of Expertscited by 1,785
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2024Qwen2.5-Coder Technical Reportcited by 1,151
38 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2025Qwen-Image Technical Reportcited by 973
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2023A Survey on Multimodal Large Language Modelscited by 898
  7. 2025Large Language Diffusion Modelscited by 872
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  9. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  10. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
  13. 2025Qwen2.5-Omni Technical Reportcited by 704
  14. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  15. 2024Phi-4 Technical Reportcited by 675
  16. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  17. 2024Self-Rewarding Language Modelscited by 654
  18. 2024A Survey on Large Language Models for Code Generationcited by 646
  19. 2023A Comprehensive Overview of Large Language Modelscited by 639
  20. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  21. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  22. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  23. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  24. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  25. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  26. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  27. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  28. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  29. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  30. 2024Qwen2-Audio Technical Reportcited by 543
  31. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  32. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  33. 2024Large Language Models: A Surveycited by 506
  34. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  35. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  36. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  37. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  38. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484

50 most cited of 7,916 All 7,916 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2024Reinforcement Learning for LLM Post-Training: A Survey
  3. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  4. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  5. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
  6. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms