mlsanityData to 7 Oct 2026

A Minimaximalist Approach to Reinforcement Learning from Human Feedback

arXiv 2401.04056 ↗First posted 8 Jan 2024cs.LGcited by 141

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement learning from human feedback. Our approach is minimalist in that it does not require training a reward model nor unstable adversarial training and is therefore rather simple to implement. Our approach is maximalist in that it provably handles non-Markovian, intransitive, and stochastic preferences while being robust to the compounding errors that plague offline approaches to sequential prediction. To achieve the preceding qualities, we build upon the concept of a Minimax Winner (MW), a notion of preference aggregation from the social choice theory literature that frames learning from preferences as a zero-sum game between two policies. By leveraging the symmetry of this game, we prove that rather than using the traditional technique of dueling two policies to compute the MW, we can simply have a single agent play against itself while maintaining strong convergence guarantees. Practically, this corresponds to sampling multiple trajectories from a policy, asking a preference or teacher model to compare them, and then using the proportion of wins as the reward for a particular trajectory. We demonstrate that on a suite of continuous control tasks, we are able to learn significantly more efficiently than reward-model based approaches while maintaining robustness to the intransitive and stochastic preferences that frequently occur in practice when aggregating human judgments.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 101 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

43 of 101 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  7. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2016OpenAI Gymcited by 1,684
  9. 2018DeepMind Control Suitecited by 935
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  11. 2019Introduction to Online Convex Optimizationcited by 793
  12. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
21 more
  1. 2017Training GANs with Optimismcited by 217
  2. 2021Reinforcement learning based recommender systems: A surveycited by 187
  3. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  4. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  5. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  6. 2015Contextual Dueling Banditscited by 101
  7. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  8. 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 97
  9. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  10. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  11. 2023Learning to Generate Better Than Your LLMcited by 49
  12. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  13. 2017Multi-dueling Bandits with Dependent Armscited by 35
  14. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 35
  15. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  16. 2022Causal Imitation Learning under Temporally Correlated Noisecited by 28
  17. 2023Uncoupled and Convergent Learning in Two-Player Zero-Sum Markov Games with Bandit Feedbackcited by 28
  18. 2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 23
  19. 2022A Ranking Game for Imitation Learningcited by 18
  20. 2019Scaled Autonomy: Enabling Human Operators to Control Robot Fleetscited by 16
  21. 2015Consistent Probabilistic Social Choicecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 59; 2025: 55; 2026: 27.

141 papers.

Sort:

By area:

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  3. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  5. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  7. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  9. 2024A Roadmap to Pluralistic Alignmentcited by 175
  10. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  11. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  12. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
38 more
  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  3. 2024Critique-out-Loud Reward Modelscited by 105
  4. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  5. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  6. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  7. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  8. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  9. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  10. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  11. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  12. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  13. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  14. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  15. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  16. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  17. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  18. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  19. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  20. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  21. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  22. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  23. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  24. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38
  25. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  26. 2024Personality Alignment of Large Language Modelscited by 36
  27. 2025A Survey of Direct Preference Optimizationcited by 35
  28. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  29. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  30. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  31. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  32. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 26
  33. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  34. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  35. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  36. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  37. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  38. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Modelscited by 20

50 most cited of 141 All 141 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  2. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  3. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization
  6. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning
  7. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
  8. 2026Learning Kernel-Based MDPs from Episodic Preferential Feedback