mlsanityData to 14 Oct 2026

A General Theoretical Paradigm to Understand Learning from Human Preferences

arXiv 2310.12036 ↗First posted 18 Oct 2023cs.AI cs.LG stat.MLcited by 912

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The prevalent deployment of learning from human preferences through reinforcement learning (RLHF) relies on two important approximations: the first assumes that pairwise preferences can be substituted with pointwise rewards. The second assumes that a reward model trained on these pointwise rewards can generalize from collected data to out-of-distribution data sampled by the policy. Recently, Direct Preference Optimisation (DPO) has been proposed as an approach that bypasses the second approximation and learn directly a policy from collected data without the reward modelling stage. However, this method still heavily relies on the first approximation. In this paper we try to gain a deeper theoretical understanding of these practical algorithms. In particular we derive a new general objective called PsiPO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations. This new general objective allows us to perform an in-depth analysis of the behavior of RLHF and DPO (as special cases of PsiPO) and to identify their potential pitfalls. We then consider another special case for PsiPO by setting Psi simply to Identity, for which we can derive an efficient optimisation procedure, prove performance guarantees and demonstrate its empirical superiority to DPO on some illustrative examples.

Key facts. Cited by 912 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 38 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

16 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2021Convex optimizationcited by 2,515
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  8. 2019A Theory of Regularized Markov Decision Processescited by 286
  9. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  10. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  11. 2020Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learningcited by 66
  12. 2020Countering Language Drift with Seeded Iterated Learningcited by 57
3 more
  1. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  2. 2016Unsupervised Pretraining for Sequence to Sequence Learningcited by 47
  3. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 360; 2025: 329; 2026: 211.

912 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  2. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  3. 2025Process Reinforcement through Implicit Rewardscited by 504
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  8. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  9. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  10. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  11. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  12. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
38 more
  1. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  2. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  3. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  4. 2023Nash Learning from Human Feedbackcited by 224
  5. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  6. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  8. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  10. 2024Generative Reward Modelscited by 148
  11. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  12. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  13. 2024Nemotron-4 340B Technical Reportcited by 144
  14. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  15. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  16. 2023Controlled Decoding from Language Modelscited by 135
  17. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  18. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  19. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  20. 2024Direct Preference Optimization with an Offsetcited by 122
  21. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  22. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  23. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
  24. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  25. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  26. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  27. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  28. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  29. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  30. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 96
  31. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  32. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  33. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  34. 2024Learning Dynamics of LLM Finetuningcited by 95
  35. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  36. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 93
  37. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  38. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90

50 most cited of 912 All 912 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Active Learning for Direct Preference Optimization
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  3. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
  4. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  5. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  6. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  7. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
  8. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs