Adversarial Training for Process Reward Models
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Process Reward Models (PRMs) enhance reasoning ability of LLMs by providing step-level supervision. However, their widespread adoption is limited due to expensive manual step-level annotation and poor generalization of static training data to novel errors. We introduce Adversarially Trained PRMs (textttAPRM), where a Generator (G) learns to produce reasoning errors to deceive a PRM (R), while R concurrently learns to detect them. This interaction yields progressively harder negatives for R, improving its robustness and generalization to novel errors without requiring manual step-level labels. Averaged across diverse mathematical reasoning benchmarks, textttAPRM improves solver accuracy by +3.4 percentage points (pp) over the strongest PRM baseline. textttAPRM achieves gains of +5.3 pp on out-of-distribution tasks.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 63 references that are in the map. First posted on arXiv 28 Nov 2025.
Builds on
33 of 63 references.
Closest ancestors · 9
Same direction, newest first
- 2025FreePRM: Training Process Reward Models Without Ground Truth Process Labels4 Jun
- 2025Process Reinforcement through Implicit Rewards3 Feb
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
- 2024Process Reward Model with Q-Value Rankings15 Oct
- 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
- 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
- 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements13 Feb
- 2023Let's Verify Step by Step31 May
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 24
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
- 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
- 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
12 more
- 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 136
- 2017Emergent Complexity via Multi-Agent Competitioncited by 120
- 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 85
- 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 77
- 2019Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Gamescited by 42
- 2014Generative Adversarial Networkscited by 33
- 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32
- 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 31
- 2024FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoningcited by 12
- 2019Game Theoretic Optimization via Gradient-based Nikaido-Isoda Functioncited by 6
- 2018Optimal Slice Allocation in 5G Core Networkscited by 2
- 2023U-SEANNet: A Simple, Efficient and Applied U-Shaped Network for Diagnosis of Nasal Diseases on Nasal Endoscopic Imagescited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewardscited by 4
- 2026Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certifycited by 0
- 2026GLARE: Generative Learning via Adversarial Reward Estimation For Social Dynamics Forecastingcited by 0
- 2026GLARE: Generative Learning via Adversarial Reward Estimation For Social Dynamics Forecasting10 Sep 2026
- 2026Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certify8 Aug 2026
- 2026Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards17 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models
- 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
- 2025R-PRM: Reasoning-Driven Process Reward Modeling
- 2025ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth
- 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning
- 2026The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
- 2026Unsupervised Process Reward Models
- 2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision