Seirenes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
We present Seirenes, a self-play RL framework that transforms contextual interference from a failure mode of LLM reasoning into an internal training signal for co-evolving more resilient reasoners. While RL with verifiable rewards has significantly advanced reasoning capabilities, models can still exhibit fragility when encountering non-idealized contexts: scenarios characterized by superfluous information, tangential instructions, or incidental correlations that differ from the clean distributions typical of standard benchmarks. Seirenes harnesses this vulnerability through a parameter-shared and adversarial self-play loop. Within this framework, a single model is trained to both construct plausible yet distracting contexts that expose its own reasoning blind spots, and solve problems by discerning the essential task from these perturbations to recover the core underlying logic. By pitting these competing objectives against each other, Seirenes compels the model to move beyond superficial pattern matching and anchors its capabilities in robust underlying reasoning. This continuous interaction sustains an informative co-evolutionary curriculum as the model improves. Across seven mathematical reasoning benchmarks and model scales from 4B to 30B, Seirenes achieves average gains of +10.2, +9.1, and +7.2 points. Besides, distracting contexts produced by the 4B Seirenes model reduce the accuracy of top-tier closed-source models (GPT and Gemini) by roughly 4--5 points, revealing Seirenes' general ability to uncover reasoning models' blind spots.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 74 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
44 of 74 references.
Closest ancestors · 16
Same direction, newest first
- 2026Learning to Hint for Reinforcement Learning1 Apr
- 2026Self-Hinting Language Models Enhance Reinforcement Learning3 Feb
- 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning20 Jan
- 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning22 Oct
- 2025ExGRPO: Learning to Reason from Experience2 Oct
- 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason3 Jul
- 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
- 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
- 2025Learning to Reason under Off-Policy Guidance21 Apr
- 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
- 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
- 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
4 more
Other references · 28
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2024OpenAI o1 System Cardcited by 1,244
- 2025Group Sequence Policy Optimizationcited by 835
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
- 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
16 more
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
- 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
- 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
- 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 89
- 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
- 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
- 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 64
- 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
- 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 47
- 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
- 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
- 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
- 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
- 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Modelscited by 13
- 2025Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Searchcited by 10
- 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Towards Understanding Self-play for LLM Reasoning
- 2026Learning Robust Reasoning through Guided Adversarial Self-Play
- 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
- 2025Better LLM Reasoning via Dual-Play
- 2024Self-playing Adversarial Language Game Enhances LLM Reasoning
- 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
- 2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play
- 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuition