mlsanityData to 7 Oct 2026

SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning

arXiv 2504.19162 ↗First posted 27 Apr 2025cs.CL cs.AI cs.LGcited by 40

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Evaluating the step-by-step reliability of large language model (LLM) reasoning, such as Chain-of-Thought, remains challenging due to the difficulty and cost of obtaining high-quality step-level supervision. In this paper, we introduce Self-Play Critic (SPC), a novel approach where a critic model evolves its ability to assess reasoning steps through adversarial self-play games, eliminating the need for manual step-level annotation. SPC involves fine-tuning two copies of a base model to play two roles, namely a "sneaky generator" that deliberately produces erroneous steps designed to be difficult to detect, and a "critic" that analyzes the correctness of reasoning steps. These two models engage in an adversarial game in which the generator aims to fool the critic, while the critic model seeks to identify the generator's errors. Using reinforcement learning based on the game outcomes, the models iteratively improve; the winner of each confrontation receives a positive reward and the loser receives a negative reward, driving continuous self-evolution. Experiments on three reasoning process benchmarks (ProcessBench, PRM800K, DeltaBench) demonstrate that our SPC progressively enhances its error detection capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and surpasses strong baselines, including distilled R1 model. Furthermore, SPC can guide the test-time search of diverse LLMs and significantly improve their mathematical reasoning performance on MATH500 and AIME2024, surpassing those guided by state-of-the-art process reward models.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 33 of its 58 references that are in the map. First posted on arXiv 27 Apr 2025.

Builds on

33 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement20 Mar

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
20 more
  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  3. 2024DeepSeek-V3 Technical Reportcited by 724
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  5. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  7. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  8. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  9. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  10. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  11. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMscited by 162
  12. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  13. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  14. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  15. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  16. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  17. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  18. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  19. 2024Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Criticcited by 27
  20. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 24.

40 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  2. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  3. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  4. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Modelscited by 61
  5. 2025TreeRPO: Tree Relative Policy Optimizationcited by 45
  6. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  7. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 31
  8. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  9. 2026G-Zero: Self-Play for Open-Ended Generation from Zero Datacited by 15
  10. 2025Lifelong Safety Alignment for Language Modelscited by 13
  11. 2025AutoCode: LLMs as Problem Setters for Competitive Programmingcited by 11
  12. 2026V-Zero: Self-Improving Multimodal Reasoning with Zero Annotationcited by 9
28 more
  1. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  2. 2025Better LLM Reasoning via Dual-Playcited by 7
  3. 2026Digital Red Queen: Adversarial Program Evolution in Core War with LLMscited by 7
  4. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 7
  5. 2026Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learningcited by 7
  6. 2025RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedbackcited by 6
  7. 2025CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewardscited by 6
  8. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6
  9. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  10. 2025CLATTER: Comprehensive Entailment Reasoning for Hallucination Detectioncited by 5
  11. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 5
  12. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  13. 2026Pseudo-Formalization for Automatic Proof Verificationcited by 3
  14. 2026STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rulescited by 2
  15. 2026iGRPO: Self-Feedback-Driven LLM Reasoningcited by 2
  16. 2025Differences That Matter: Auditing Models for Capability Gap Discovery and Rectificationcited by 1
  17. 2026AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedbackcited by 1
  18. 2026TTSR: Test-Time Self-Evolving via Reflectioncited by 1
  19. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
  20. 2026R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoningcited by 0
  21. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  22. 2026Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Playcited by 0
  23. 2026ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoningcited by 0
  24. 2026Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generatorcited by 0
  25. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvementcited by 0
  26. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  27. 2026STRETCH the Boundaries: A Unified Self-Taught Framework for Progressive LLM Evolutioncited by 0
  28. 2026Information-Gain Rewards over Diversity-Pruned Tests: GT-Anchored Verifier Co-Training for Reliable Code Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-playing Adversarial Language Game Enhances LLM Reasoning
  2. 2025Towards Understanding Self-play for LLM Reasoning
  3. 2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
  4. 2026Self-Play Search Distillation for Large Language Model Reasoning
  5. 2025Enhancing Language Agent Strategic Reasoning through Self-Play in Adversarial Games
  6. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  7. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution
  8. 2026Seirenes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning