Grading the Graders: Verification Autonomy Levels (L0-L5) for LLM Reasoning
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Large language models (LLMs) are increasingly paired with verifiers (step checkers, self-consistency filters, tool-based fact checkers, formal proof assistants) that claim to detect the model's errors. Yet the verification literature uses the word "level" to mean at least five different things: verification granularity, concept abstraction, risk tier, system-stack layer, and the epistemic source of the ground truth. We propose Verification Autonomy Levels (VAL), a meta-standard that classifies any verification scheme along a single axis: where does the verification spec come from, and what does the verdict guarantee? VAL ranges from L0 (LLM self-declaration; no deterministic anchor) through L2 (objective ground truth; correctness only) to L3/L4 (decidable systems with single-property or domain-level completeness), with L5 impossible in the unrestricted case. Central to VAL is the completeness blind spot: substitution- and sampling-based verifiers can confirm that proposed candidates hold, but cannot prove that no candidate was missed. We further identify a dichotomy the literature has not stated: completeness is reachable only for formally specifiable properties, whereas empirical open-world verification (fact-checking, diagnosis) caps at anchored correctness (L2). We document this gap empirically across four domains (symbolic mathematics, behavior monitoring, medical diagnosis, code generation) and in the strongest formal-verification baseline in our survey. We show that granularity, concept hierarchy, risk, and system stack are orthogonal to VAL, resolving a conflation across 17 surveyed papers. Code and the full literature assessment are released on Zenodo (DOI: 10.5281/zenodo.23120985).
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 11 of its 13 references that are in the map. First posted on arXiv 19 Aug 2026.
Builds on
11 of 13 references.
Closest ancestors · 3
Same direction, newest first
- 2025VerifiAgent: a Unified Verification Agent in Language Model Reasoning1 Apr
- 2024textttLM^texttt2: A Simple Society of Language Models Solves Complex Reasoning2 Apr
- 2023Beyond Words: A Mathematical Framework for Interpreting Large Language Models6 Nov
Other references · 8
- 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
- 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
- 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
- 2023Prompting Frameworks for Large Language Models: A Surveycited by 29
- 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
- 2025LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflowscited by 6
- 2025DeepKnown-Guard: A Proprietary Model-Based Safety Response Framework for AI Agentscited by 2
- 2025Hierarchical Attention Generates Better Proofscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Beyond Correctness: Exposing LLM-generated Logical Flaws in Reasoning via Multi-step Automated Theorem Proving
- 2025Variation in Verification: Understanding Verification Dynamics in Large Language Models
- 2026A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- 2026LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
- 2026VERGE: Formal Refinement and Guidance Engine for Verifiable LLM Reasoning
- 2026Verification as an Architectural Layer for LLM Agents: A V-Model Design, and a Pilot Study of Its Deterministic Core
- 2026Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism
- 2024Learning From Correctness Without Prompting Makes LLM Efficient Reasoner