mlsanityData to 7 Oct 2026

Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play

arXiv 2411.00062 ↗First posted 31 Oct 2024cs.CL cs.AI physics.data-ancited by 21

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Current reinforcement learning (RL) frameworks for large language models (LLM) post-training typically assume a fixed prompt distribution, which is sub-optimal and bottlenecks scalability. Prior works have explored prompt evolving, but are often limited to the supervised fine-tuning stage, and prompts are sampled and evolved uniformly without signals. This empirical work presents a paradigm shift: Evolving Alignment via Asymmetric Self-Play (eva), that casts post-training as an infinite game with regret-based signals for 2 players: (i) a creator, who strategically samples and creates new informative prompts and (ii) a solver, who learns to produce preferred responses. eva is the first method that allows language models to adaptively create training prompts in both offline and online RL post-training. The design is simple, easy-to-use yet remarkably effective: eva sets a new SOTA on challenging benchmarks, without any extra human prompts, e.g. it boosts the win-rate of gemma-2-9b-it on Arena-Hard by 51.6% -> 60.1% for DPO and 52.6% -> 62.4% for RLOO, surpassing claude-3-opus and catching up to gemini-1.5-pro, both of which are orders of magnitude larger. Extensive experiments show eva can create effective RL curricula and is robust across ablations. We believe adaptively evolving prompts are key to designing the next-generation RL post-training scheme.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 73 of its 106 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

73 of 106 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  2. 2024Toward Optimal LLM Alignments Using Two-Player Games16 Jun
  3. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  4. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  5. 2024Self-Rewarding Language Models18 Jan
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  7. 2023Nash Learning from Human Feedback1 Dec
  8. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  9. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  10. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr

Other references · 61

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2024Genie: Generative Interactive Environmentscited by 706
38 more
  1. 2022Large Language Models Can Self-Improvecited by 656
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  3. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  4. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  7. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  8. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  9. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  12. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  13. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  14. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  15. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  16. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  17. 2024Iterative Reasoning Preference Optimizationcited by 244
  18. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  19. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  20. 2020Prioritized Level Replaycited by 190
  21. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  22. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  23. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  24. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  25. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  26. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  27. 2021Replay-Guided Adversarial Environment Designcited by 108
  28. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  29. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  30. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  31. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  32. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  33. 2024Active Preference Learning for Large Language Modelscited by 55
  34. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51
  35. 2024Beyond Preferences in AI Alignmentcited by 46
  36. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  37. 2024Learning Formal Mathematics From Intrinsic Motivationcited by 45
  38. 2024Efficient Exploration for LLMscited by 44

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 10.

21 papers.

Sort:

By area:

  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  2. 2025STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Provingcited by 77
  3. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 58
  4. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
  5. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
  6. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  7. 2025Lifelong Safety Alignment for Language Modelscited by 13
  8. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  9. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
  10. 2025Game Theory Meets LLM and Agentic AI: Reimagining Cybersecurity for the Age of Intelligent Threatscited by 10
  11. 2025AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Playcited by 10
  12. 2025The Alignment Waltz: Jointly Training Agents to Collaborate for Safetycited by 10
9 more
  1. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 7
  2. 2026GASP: Guided Asymmetric Self-Play For Coding LLMscited by 4
  3. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 4
  4. 2026iGRPO: Self-Feedback-Driven LLM Reasoningcited by 2
  5. 2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Playcited by 2
  6. 2026LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RLcited by 0
  7. 2026Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimizationcited by 0
  8. 2026Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Modelscited by 0
  9. 2026False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SEIF: Self-Evolving Reinforcement Learning for Instruction Following
  2. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  3. 2026Demystifying Reinforcement Learning Post-Training of Language Models
  4. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
  5. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
  6. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
  7. 2026Evolutionary System Prompt Learning for Reinforcement Learning in LLMs
  8. 2025PARL: Prompt-based Agents for Reinforcement Learning