mlsanityData to 14 Oct 2026

PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play

arXiv 2605.16727 ↗First posted 16 May 2026cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

We introduce PopuLoRA, a population-based asymmetric self-play framework for reinforcement learning with verifiable rewards (RLVR) post-training of LLMs. Teachers and students are specialised LoRA adapters on a shared frozen base: teachers propose problems, matched students solve them under a programmatic verifier, and cross-evaluation between sub-populations replaces the self-calibration that limits single-agent self-play. A family of LoRA weight-space evolution operators (mutations and crossovers that produce same-rank population members in seconds) serves as the replacement step of a population-based training loop at 7B scale. We instantiate PopuLoRA on top of Absolute Zero Reasoner and compare it against a per-adapter compute-matched single-agent baseline. Where the single agent self-calibrates to generating easy problems it can reliably solve, the population enters a co-evolutionary arms race: teachers produce increasingly complex problems, student solve rates oscillate, and problem-space coverage keeps expanding throughout training. Despite lower training-time reward, the population mean outperforms the baseline on three code benchmarks (HumanEval+, MBPP+, LiveCodeBench) and seven math benchmarks (AIME 24/25, AMC 23, MATH-500, Minerva, GSM8K, OlympiadBench), and even the weakest member of the population beats the baseline on aggregate.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 62 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

27 of 62 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluation5 Feb
  2. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability26 Jan
  3. 2025ESSA: Evolutionary Strategies for Scalable Alignment6 Jul
  4. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  5. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  7. 2017Population Based Training of Neural Networkscited by 367
  8. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  9. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  10. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  11. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  12. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
8 more
  1. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  2. 2025Language Self-Play For Data-Free Trainingcited by 43
  3. 2024X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Designcited by 42
  4. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 34
  5. 2025Evolution Strategies at the Hyperscalecited by 25
  6. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  7. 2025Nature-Inspired Population-Based Evolution of Large Language Modelscited by 15
  8. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignmentcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 35
  2. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution
  2. 2025Towards Understanding Self-play for LLM Reasoning
  3. 2026SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning
  4. 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoning
  5. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills
  6. 2026EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning
  7. 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
  8. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning