Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap
On the map: LLM agents and tool use → Tool learning and function calling
Abstract (arXiv)
Small open-source large language models (LLMs) in the 0.6B-4B parameter range are increasingly deployed for structured output generation (JSON, function calling, data extraction), yet little is known about how constrained decoding (CD) interacts with model scale in this regime. We benchmark five models from three families across 14 structured-output tasks under three decoding conditions (native, Outlines, XGrammar). We introduce a two-axis evaluation that separates structural correctness (schema validity) from semantic correctness (content accuracy). We find that CD eliminates all structural failures across all models (schema validity: 78.6-92.9% to 100%), but content accuracy reveals a persistent semantic gap that is scale-dependent: type coercion failures are fully CD-rescuable, while instruction-semantic failures (e.g., multi-step function calling) remain CD-resistant. Schema conformance is necessary but not sufficient for semantic correctness; CD's reach ends exactly where schema conformance ends.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 5 of its 16 references that are in the map. First posted on arXiv 20 Sep 2026.
Builds on
5 of 16 references.
References · 5
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
- 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 66
- 2026The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decodingcited by 7
- 2025STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliabilitycited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026From Hallucination to Structure Snowballing: The Alignment Tax of Constrained Decoding in LLM Reflection
- 2026Thinking Before Constraining: A Unified Decoding Framework for Large Language Models
- 2025Constrained Decoding of Diffusion LLMs with Context-Free Grammars
- 2025On the Fundamental Limits of LLMs at Scale
- 2026The Alignment Problem in Constrained Code Generation
- 2025On Evaluating Performance of LLM Inference Serving Systems
- 2025Semantic uncertainty in advanced decoding methods for LLM generation
- 2026Where vs What: Decomposing Structural and Content Failures in LLM-Generated Structured Outputs