Capacity, Not Format: Rethinking Structured Reasoning Failures
Not on the map. Why?
Abstract (arXiv)
Prior work treats structured output as a reasoning tax, but this framing is incomplete: the cost of formatting depends strongly on a model's spare capacity. Using information-matched prose controls and a four-level schema complexity gradient, we separate format-specific effects from prompt-length confounds across 4 models and 5 benchmarks with 0% parse failures on successfully generated responses. We find that structured formats are capacity-dependent. Models with sufficient headroom absorb JSON constraints without degradation (Sonnet: 88.7±4.0% JSON vs. 89.3±1.7% CoT on MATH-Hard). In contrast, formats severely degrade models operating near their limits through two distinct mechanisms. First, under standard token budgets, Haiku drops 36.2pp (p < 0.0001) largely due to truncation. Second, even with extended budgets eliminating truncation, GPT-4o-mini drops 28.0pp (p < 0.001), revealing pure capacity competition independent of token exhaustion. This format penalty scales with schema complexity (McNemar p < 0.0001) and cannot be explained by prompt length alone. Furthermore, these results qualify claims of frontier model immunity: on AIME competition math, Opus 4.7 drops from 96.2% to 91.0% under JSON (-5.3pp; the displayed percentages are independently rounded, exact difference is 7/133 = 5.26pp ≈ 5.3pp). A delayed-structure ablation -- reasoning freely before formatting -- recovers most of the lost accuracy (3-run mean: 80--87%), supporting the capacity competition mechanism. The practical implication is not to avoid structured output, but to match it to capacity: when a model is near its limits, think first, format later.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 6 of its 17 references that are in the map. First posted on arXiv 8 Jun 2026.
Builds on
6 of 17 references.
References · 6
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2025Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inferencecited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models
- 2026Where vs What: Decomposing Structural and Content Failures in LLM-Generated Structured Outputs
- 2026Breaking the Space Barrier and its Application to Language Model Inference
- 2025Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models
- 2025How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach
- 2026Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap
- 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression
- 2026The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits