The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale
Not on the map. Why?
Abstract (arXiv)
Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning. We show this can fail at the answer-extraction boundary, and test the failure causally rather than only observationally. Across Qwen3.5 (0.8B-9B), Gemma-4-12B, and two frontier models via API (Tencent Hy3, Nvidia Nemotron-3-Ultra-550B) in 29 primary cells plus a frontier arm, we decompose the always-revise accuracy shift into a content margin (both answers parseable) and format-recovery/loss margins (parseability changes). On 12 cells with meaningful unparseable-answer rates, format effects exceed content effects (Wilcoxon p=1.7e-3). To test this causally, we force already-generated reasoning through grammar-constrained decoding so every answer is parseable by construction: across 14 cells this closes a median 71% of the gap between the naive total effect and the content-margin estimate, with two cells converging exactly and a residual on the two largest-effect cells reported rather than dismissed. A clustered model confirms floor-scale (0.8B/2B) models have far higher odds of content-level change and harm than capable-scale models (p<1e-7). Replicating a cited confidence-gating protocol verbatim on Qwen3.5 does not reproduce its reported gain and shows the same near-zero content margin. A frontier check on much larger models shows format-dominance intensifying with scale: content margin is exactly zero in all 5 cells despite total effects up to +0.275, though this arm is lower-powered. The calibration-floor criterion on the content margin reveals a squeeze: floor-scale cells have headroom but insufficient signal, capable-scale cells have signal but little headroom; only one cell is marginally viable, with negligible sealed-holdout gain. Content is a minority share of what the field has measured as self-correction. We release the instrument, code, and derived results.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 5 Aug 2026.
Builds on
23 of 31 references.
References · 23
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
11 more
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
- 2024Self-Rewarding Language Modelscited by 654
- 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
- 2026Qwen3.5-Omni Technical Reportcited by 146
- 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
- 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 25
- 2026When Does Intrinsic Self-Correction Help? A Task-Sensitive Analysiscited by 3
- 2026When are likely answers right? On Sequence Probability and Correctness in LLMscited by 3
Built on by
Not cited yet.