mlsanityData to 7 Oct 2026

Self-Play Preference Optimization for Language Model Alignment

arXiv 2405.00675 ↗First posted 1 May 2024cs.LG cs.AI cs.CLcited by 228

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Standard reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game aimed at identifying the Nash equilibrium policy. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium. Additionally, we propose a new SPPO objective which is both strongly motivated by theory and is simple and effective in practice. In our experiments, using only 60k prompts (without responses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench, Arena-Hard, and the Open LLM Leaderboard. Starting from a stronger base model Llama-3-8B-Instruct, we are able to achieve a length-controlled win rate of 38.77%. Notably, the strong performance of SPPO is achieved without additional external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models. Codes are available at https://github.com/uclaml/SPPO.

Key facts. Cited by 228 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 59 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

39 of 59 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024REBEL: Reinforcement Learning via Regressing Relative Rewards25 Apr
  2. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  3. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  4. 2024Self-Rewarding Language Models18 Jan
  5. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  6. 2023Nash Learning from Human Feedback1 Dec
  7. 2023Is RLHF More Difficult than Standard RL?25 Jun
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  10. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2023Mistral 7Bcited by 3,065
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
15 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  6. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  7. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  8. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  9. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  10. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  11. 2023Generative Judge for Evaluating Alignmentcited by 163
  12. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  13. 2015Contextual Dueling Banditscited by 101
  14. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  15. 2023Borda Regret Minimization for Generalized Linear Dueling Banditscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 76; 2025: 93; 2026: 59.

228 papers.

Sort:

By area:

  1. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  4. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  5. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  6. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  7. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  8. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  9. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  10. 2024Learning Dynamics of LLM Finetuningcited by 93
  11. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  12. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
38 more
  1. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  2. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  3. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  4. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  5. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  6. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  7. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  8. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  9. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  10. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  11. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  12. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  13. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  14. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  15. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  16. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  17. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  18. 2025Language Self-Play For Data-Free Trainingcited by 42
  19. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  20. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
  21. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 39
  22. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  23. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  24. 2025A Survey of Direct Preference Optimizationcited by 35
  25. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  26. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  27. 2024Reward-Robust RLHF in LLMscited by 32
  28. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  29. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 30
  30. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  31. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  32. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  33. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  34. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reusecited by 28
  35. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignmentcited by 27
  36. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  37. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  38. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24

50 most cited of 228 All 228 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  2. 2024Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models
  3. 2024Investigating Regularization of Self-Play Language Models
  4. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  7. 2024Towards Efficient Exact Optimization of Language Model Alignment
  8. 2026Spectral Souping: A Unified Framework for Online Preference Alignment