# Toward Training Superintelligent Software Agents through Self-Play SWE-RL

arXiv: https://arxiv.org/abs/2512.18552 · HTML: https://mlsanity.com/paper/2512.18552/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 57 references that are in the map.
- First posted on arXiv 21 Dec 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 22 of 22)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025)
- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)
- [Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?](https://mlsanity.com/paper/2511.13646/) (2025)
- [CWM: An Open-Weights LLM for Research on Code Generation with World Models](https://mlsanity.com/paper/2510.02387/) (2025)
- [SPICE: Self-Play In Corpus Environments Improves Reasoning](https://mlsanity.com/paper/2510.24684/) (2025)
- [Huxley-G"odel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine](https://mlsanity.com/paper/2510.21614/) (2025)
- [Self-Questioning Language Models](https://mlsanity.com/paper/2508.03682/) (2025)
- [Language Self-Play For Data-Free Training](https://mlsanity.com/paper/2509.07414/) (2025)
- [Kimi-Dev: Agentless Training as Skill Prior for SWE-Agents](https://mlsanity.com/paper/2509.23045/) (2025)
- [Stabilizing Reinforcement Learning with LLMs: Formulation and Practices](https://mlsanity.com/paper/2512.01374/) (2025)
- [Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation](https://mlsanity.com/paper/2502.14948/) (2025)
- [BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills](https://mlsanity.com/paper/2510.19898/) (2025)
- [CodeClash: Benchmarking Goal-Oriented Software Engineering](https://mlsanity.com/paper/2511.00839/) (2025)

## Built on by (most cited first, 30 of 40)

- [Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharing](https://mlsanity.com/paper/2602.04837/) (2026)
- [SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving](https://mlsanity.com/paper/2601.01426/) (2026)
- [SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale](https://mlsanity.com/paper/2602.23866/) (2026)
- [Agentic Rubrics as Contextual Verifiers for SWE Agents](https://mlsanity.com/paper/2601.04171/) (2026)
- [Position: Agentic Evolution is the Path to Evolving LLMs](https://mlsanity.com/paper/2602.00359/) (2026)
- [CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution](https://mlsanity.com/paper/2603.17775/) (2026)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills](https://mlsanity.com/paper/2606.07412/) (2026)
- [Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis](https://mlsanity.com/paper/2605.14392/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Measuring all the noises of LLM Evals](https://mlsanity.com/paper/2512.21326/) (2025)
- [Digital Red Queen: Adversarial Program Evolution in Core War with LLMs](https://mlsanity.com/paper/2601.03335/) (2026)
- [Learning to Repair Lean Proofs from Compiler Feedback](https://mlsanity.com/paper/2602.02990/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents](https://mlsanity.com/paper/2602.10356/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [Hindsight Hint Distillation: Scaffolded Reasoning for SWE Agents from CoT-free Answers](https://mlsanity.com/paper/2605.11556/) (2026)
- [SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Tasks](https://mlsanity.com/paper/2603.00575/) (2026)
- [FrogNano: Training a 4B Coding Agent via Online Task Synthesis](https://mlsanity.com/paper/2609.07925/) (2026)
- [PREPING: Zero-Task Agent Memory Construction via Self-Practice](https://mlsanity.com/paper/2605.13880/) (2026)
- [Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier](https://mlsanity.com/paper/2606.18284/) (2026)
- [Yet Even Less Is Even Better For Agentic, Reasoning, and Coding LLMs](https://mlsanity.com/paper/2604.00824/) (2026)
- [Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models](https://mlsanity.com/paper/2607.12463/) (2026)
- [PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents](https://mlsanity.com/paper/2607.18859/) (2026)
- [Self-Evolving Coding Agents](https://mlsanity.com/paper/2608.03392/) (2026)
- [Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First](https://mlsanity.com/paper/2608.04804/) (2026)
- [Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning](https://mlsanity.com/paper/2606.11719/) (2026)
- [Anchored Self-Play for Code Repair](https://mlsanity.com/paper/2607.03523/) (2026)

## Abstract (arXiv)

While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.g., GitHub issues and pull requests) and environments (e.g., pass-to-pass and fail-to-pass tests) heavily depend on human knowledge or curation, posing a fundamental barrier to superintelligence. In this paper, we present Self-play SWE-RL (SSR), a first step toward training paradigms for superintelligent software agents. Our approach takes minimal data assumptions, only requiring access to sandboxed repositories with source code and installed dependencies, with no need for human-labeled issues or tests. Grounded in these real-world codebases, a single LLM agent is trained via reinforcement learning in a self-play setting to iteratively inject and repair software bugs of increasing complexity, with each bug formally specified by a test patch rather than a natural language issue description. On the SWE-bench Verified and SWE-Bench Pro benchmarks, SSR achieves notable self-improvement (+10.4 and +7.8 points, respectively) and consistently outperforms the human-data baseline over the entire training trajectory, despite being evaluated on natural language issues absent from self-play. Our results, albeit early, suggest a path where agents autonomously gather extensive learning experiences from real-world software repositories, ultimately enabling superintelligent systems that exceed human capabilities in understanding how systems are constructed, solving novel challenges, and autonomously creating new software from scratch.