Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Frontier gains in language-model reasoning come from reinforcement learning on reasoning traces and are concentrated in domains with a cheap, sound verifier. We argue the field's binding constraint is the verification gap: no scalable, incorruptible reward for reasoning outside formal domains. We make four contributions. (1) Theory: in a joint-Gaussian model of best-of-N selection, verifier-gold correlation rho is the exact exchange rate between test-time compute and capability, and an unsound verifier pays a polynomial penalty N^(1/rho^2); a margin-free copula form predicts realized soundness of real LLM judges to 4% median error. (2) Demonstration: in program-synthesis testbeds with executable ground truth, including a pre-registered scaled replication, unsound verifiers lose Soundness-under-Pressure as optimization grows (0.94 to 0.32 at N=4096) while a sound verifier improves monotonically; reality-anchored settlement beats a frozen verifier under i.i.d. and adversarial pressure, driving the hacking gap from ~0.27 to ~0; soundness scales log-linearly with settled labels, with on-policy settlement ~10x more label-efficient than random labeling. With real LLM judges and unit-test execution as gold, a weak judge loses soundness under best-of-N (p<0.001), a stronger judge is more robust, and selection alone manufactures +0.53 hacking gaps from honest samples. Under real GRPO training, a frozen reward model traces the full overoptimization curve (executed reward collapses 90%) while the same model refit on a 10% settlement stream preserves 6x the executed reward. (3) Paradigm: proof-carrying cognition, where reasoning steps are typed probabilistic claims priced by a self-built world model trained only on held-out reality and settled by proper scoring rules. (4) Benchmark: we specify Soundness-under-Pressure as the headline metric for a reality-settled reasoning benchmark.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 35 references that are in the map. First posted on arXiv 9 Sep 2026.
Builds on
21 of 35 references.
Closest ancestors · 2
Same direction, newest first
- 2023Let's Verify Step by Step31 May
- 2022Solving math word problems with process- and outcome-based feedback25 Nov
Other references · 19
- 2017Attention Is All You Needcited by 59,932
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2016Concrete Problems in AI Safetycited by 1,582
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
7 more
- 2018Scalable agent alignment via reward modeling: a research directioncited by 438
- 2018AI safety via debatecited by 384
- 2022Autoformalization with Large Language Modelscited by 252
- 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
- 2024Approaching Human-Level Forecasting with Language Modelscited by 102
- 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
- 2016Logical Inductioncited by 15
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains
- 2026Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
- 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
- 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
- 2026Verify to Amplify: Improving Reasoning via Learned Chain-of-Thought Verification
- 2026Likelihood-Based Reward Designs for General LLM Reasoning
- 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
- 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewards