# Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models

arXiv: https://arxiv.org/abs/2605.12519 · HTML: https://mlsanity.com/paper/2605.12519/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [Olmo 3](https://mlsanity.com/paper/2512.13961/) (2025)
- [DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning](https://mlsanity.com/paper/2511.22570/) (2025)
- [The Chess Transformer: Mastering Play using Generative Language Models](https://mlsanity.com/paper/2008.04057/) (2020)
- [TextArena](https://mlsanity.com/paper/2504.11442/) (2025)
- [PokeLLMon: A Human-Parity Agent for Pokemon Battles with Large Language Models](https://mlsanity.com/paper/2402.01118/) (2024)
- [Large Language Models as Minecraft Agents](https://mlsanity.com/paper/2402.08392/) (2024)
- [MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games](https://mlsanity.com/paper/2603.09022/) (2026)
- [Learning to Generate Formally Verifiable Step-by-Step Logic Reasoning via Structured Formal Intermediaries](https://mlsanity.com/paper/2603.29500/) (2026)

## Built on by (most cited first, 4 of 4)

- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?](https://mlsanity.com/paper/2609.39909/) (2026)
- [Representation-Aligned Auxiliary Supervision for Language Model Adaptation](https://mlsanity.com/paper/2610.04098/) (2026)
- [DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter](https://mlsanity.com/paper/2610.17666/) (2026)

## Abstract (arXiv)

Training language models to produce both correct answers and sound reasoning remains an open challenge. Reinforcement learning with verifiable rewards typically optimizes only final outcomes, which can improve task accuracy at the expense of reasoning quality, producing inaccurate, incomplete, or inconsistent traces. We propose verifiable process supervision (VPS), a post-training framework that jointly optimizes prediction accuracy and reasoning quality by supervising structured intermediate claims. We first apply supervised fine-tuning to induce a structured reasoning format, enabling deterministic extraction and verification of intermediate claims for process-level rewards. To address the heterogeneous difficulty of reasoning subtasks, we introduce adaptive weighting that prioritizes components with the largest remaining errors, creating an implicit curriculum. We evaluate VPS on chess as a controlled testbed where reasoning steps can be deterministically verified against engine signals. While outcome-only RL improves move accuracy, it sharply degrades reasoning quality, increasing win-rate error by up to 112% and reducing internal consistency by up to 69%. In contrast, VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation. A reasoning-space analysis further shows that, without a structured prior, outcome-only RL converges to budget-dependent shortcuts rather than sound multi-step reasoning. Beyond chess, we observe the same phenomenon on math reasoning, where outcome-only RL improves accuracy while degrading step-level arithmetic and consistency, whereas VPS maintains both. These results show that VPS enables language models to reason both accurately and reliably in verifiable domains.