mlsanityData to 14 Oct 2026

Better LLM Reasoning via Dual-Play

arXiv 2511.11881 ↗First posted 14 Nov 2025cs.LG cs.AI cs.CLcited by 8

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable progress through Reinforcement Learning with Verifiable Rewards (RLVR), yet still rely heavily on external supervision (e.g., curated labels). Adversarial learning, particularly through self-play, offers a promising alternative that enables models to iteratively learn from themselves - thus reducing reliance on external supervision. Dual-play extends adversarial learning by assigning specialized roles to two models and training them against each other, fostering sustained competition and mutual evolution. Despite its promise, adapting dual-play training to LLMs remains limited, largely due to their susceptibility to reward hacking and training instability. In this paper, we introduce PasoDoble, a novel LLM dual-play framework. PasoDoble adversarially trains two models initialized from the same base model: a Proposer, which generates challenging questions with ground-truth answers, and a Solver, which attempts to solve them. We enrich the Proposer with knowledge from a pre-training dataset to ensure the questions' quality and diversity. To avoid reward hacking, the Proposer is rewarded for producing only valid questions that push the Solver's limit, while the Solver is rewarded for solving them correctly, and both are updated jointly. To further enhance training stability, we introduce an optional offline paradigm that decouples Proposer and Solver updates, alternately updating each for several steps while holding the other fixed. Notably, PasoDoble operates without supervision during training. Experimental results show that PasoDoble can improve the reasoning performance of LLMs. Our project page is available at https://hcy123902.github.io/PasoDoble.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 35 references that are in the map. First posted on arXiv 14 Nov 2025.

Builds on

29 of 35 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  2. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  3. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  4. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  5. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models5 Jun
  6. 2025Maximizing Confidence Alone Improves Reasoning28 May
  7. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data25 May
  8. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  9. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  10. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  11. 2023Let's Verify Step by Step31 May
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2017Robust Adversarial Reinforcement Learningcited by 451
  9. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  10. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  11. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  12. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 75
5 more
  1. 2017Adversarial Feature Matching for Text Generationcited by 58
  2. 2025Self-Questioning Language Modelscited by 50
  3. 2016Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classificationcited by 48
  4. 2025Language Self-Play For Data-Free Trainingcited by 43
  5. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  2. 2026Escher-Loop: Mutual Evolution by Closed-Loop Self-Referential Optimizationcited by 8
  3. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  4. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  5. 2026GASP: Guided Asymmetric Self-Play For Coding LLMscited by 4
  6. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
  7. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  8. 2026CompassPlay: Rewarding the Proposer for Where It Moves the Solvercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Understanding Self-play for LLM Reasoning
  2. 2026Seirenes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning
  3. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models
  4. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
  5. 2025Adversarial Testing in LLMs: Insights into Decision-Making Vulnerabilities
  6. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
  7. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
  8. 2024Toward Optimal LLM Alignments Using Two-Player Games