mlsanityData to 14 Oct 2026

RSPO: Regularized Self-Play Alignment of Large Language Models

arXiv 2503.00030 ↗First posted 24 Feb 2025cs.LG cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Self-play-based policy optimization has emerged as an effective approach for fine-tuning large language models (LLMs), formulating preference optimization as a two-player game. However, the regularization with respect to the reference policy, which is crucial for mitigating over-optimization, has been insufficiently investigated in self-play alignment. To study the impact of different regularization strategies, we propose Regularized Self-Play Policy Optimization (RSPO), a novel framework that unifies prior methods and enables simple plug-and-play regularizers, meanwhile preserving convergence to Nash equilibrium of the corresponding regularized game. We empirically show that RSPO with appropriate regularizers can substantially improve the length-controlled win rate (LCWR) on AlpacaEval-2 across a range of base models, while also achieving consistently superior performance on Arena-Hard, MT-Bench, ArmoRM, and response diversity. In particular, RSPO improves unregularized self-play baseline (SPPO) on AlpacaEval-2 LCWR from 28.5% to 35.4% with base model Mistral-7B, from 38.77% to 43.66% with LLaMA-8B, and from 50.54% to 51.83% with Gemma-2B. Combining simplicity, convergence guarantees, and significant empirical gains, RSPO offers a strong foundation for exploring regularized self-play in alignment. Code is available at https://github.com/xiaohangt/RSPO

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 71 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

56 of 71 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment22 Oct
  2. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment3 Oct
  3. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  5. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  8. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  9. 2024REBEL: Reinforcement Learning via Regressing Relative Rewards25 Apr
  10. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  11. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  12. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
5 more
  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  2. 2023Nash Learning from Human Feedback1 Dec
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2022Generative Adversarial Networkscited by 5,627
  7. 2023Mistral 7Bcited by 3,164
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  12. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
27 more
  1. 2017Robust Adversarial Reinforcement Learningcited by 451
  2. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  3. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  4. 2024Iterative Reasoning Preference Optimizationcited by 246
  5. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  6. 2022Diffusion-GAN: Training GANs with Diffusioncited by 205
  7. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  9. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  10. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  11. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  12. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  13. 2020Mirror Descent Policy Optimizationcited by 106
  14. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  15. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  16. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  17. 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 63
  18. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
  19. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  20. 2019Practical and Consistent Estimation of f-Divergencescited by 24
  21. 2022The Power of Regularization in Solving Extensive-Form Gamescited by 21
  22. 2019Neural Replicator Dynamicscited by 13
  23. 2022Last-Iterate Convergence with Full and Noisy Feedback in Two-Player Zero-Sum Gamescited by 9
  24. 2023Adaptively Perturbed Mirror Descent for Learning in Gamescited by 9
  25. 2022Symmetric (Optimistic) Natural Policy Gradient for Multi-agent Learning with Parameter Convergencecited by 8
  26. 2024Boosting Perturbed Gradient Ascent for Last-Iterate Convergence in Gamescited by 5
  27. 2024Textual Aesthetics in Large Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 6
  2. 2025Proximal Point Nash Learning from Human Feedbackcited by 5
  3. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  4. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  5. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating Regularization of Self-Play Language Models
  2. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment
  3. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  4. 2025AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
  5. 2024Language Alignment via Nash-learning and Adaptive feedback
  6. 2026S-SPPO: Semantic-Calibrated Self-Play Preference Optimization
  7. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  8. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences