# Learning Robust Reasoning through Guided Adversarial Self-Play

arXiv: https://arxiv.org/abs/2602.00173 · HTML: https://mlsanity.com/paper/2602.00173/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [Geometric-Mean Policy Optimization](https://mlsanity.com/paper/2507.20673/) (2025)
- [Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning](https://mlsanity.com/paper/2507.00432/) (2025)
- [RLPR: Extrapolating RLVR to General Domains without Verifiers](https://mlsanity.com/paper/2506.18254/) (2025)
- [Self-playing Adversarial Language Game Enhances LLM Reasoning](https://mlsanity.com/paper/2404.10642/) (2024)
- [MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation](https://mlsanity.com/paper/2312.17080/) (2023)
- [The Path Not Taken: RLVR Provably Learns Off the Principals](https://mlsanity.com/paper/2511.08567/) (2025)
- [Understanding Aha Moments: from External Observations to Internal Mechanisms](https://mlsanity.com/paper/2504.02956/) (2025)
- [Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs](https://mlsanity.com/paper/2510.16062/) (2025)
- [ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning](https://mlsanity.com/paper/2507.02834/) (2025)
- [Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs](https://mlsanity.com/paper/2601.08198/) (2026)
- [Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?](https://mlsanity.com/paper/2510.06410/) (2025)

## Built on by (most cited first, 8 of 8)

- [Training-Free Looped Transformers](https://mlsanity.com/paper/2605.23872/) (2026)
- [PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planning](https://mlsanity.com/paper/2602.13691/) (2026)
- [Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning](https://mlsanity.com/paper/2602.12405/) (2026)
- [φ-Balancing for Mixture-of-Experts Training](https://mlsanity.com/paper/2605.15403/) (2026)
- [Momentum Guidance: Plug-and-Play Guidance for Flow Models](https://mlsanity.com/paper/2602.20360/) (2026)
- [AgentSpec: Understanding Embodied Agent Scaffolds Through Controlled Composition](https://mlsanity.com/paper/2606.14674/) (2026)
- [Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces](https://mlsanity.com/paper/2604.11996/) (2026)
- [Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RL](https://mlsanity.com/paper/2610.11128/) (2026)

## Abstract (arXiv)

Reinforcement learning from verifiable rewards (RLVR) produces strong reasoning models, yet they can fail catastrophically when the conditioning context is fallible (e.g., corrupted chain-of-thought, misleading partial solutions, or mild input perturbations), since standard RLVR optimizes final-answer correctness only under clean conditioning. We introduce GASP (Guided Adversarial Self-Play), a robustification method that explicitly trains detect-and-repair capabilities using only outcome verification. Without human labels or external teachers, GASP forms an adversarial self-play game within a single model: a polluter learns to induce failure via locally coherent corruptions, while an agent learns to diagnose and recover under the same corrupted conditioning. To address the scarcity of successful recoveries early in training, we propose in-distribution repair guidance, an imitation term on self-generated repairs that increases recovery probability while preserving previously acquired capabilities. Across four open-weight models (1.5B--8B), GASP transforms strong-but-brittle reasoners into robust ones that withstand misleading and perturbed context while often improving clean accuracy. Further analysis shows that adversarial corruptions induce an effective curriculum, and in-distribution guidance enables rapid recovery learning with minimal representational drift.