# Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention

arXiv: https://arxiv.org/abs/2608.13959 · HTML: https://mlsanity.com/paper/2608.13959/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 14 Aug 2026.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 12 of 12)

- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo](https://mlsanity.com/paper/2504.13139/) (2025)
- [Preregistering NLP Research](https://mlsanity.com/paper/2103.06944/) (2021)
- [Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models](https://mlsanity.com/paper/2601.05366/) (2026)
- [Mitigating Bias in Locally Constrained Decoding via Tractable Proposals](https://mlsanity.com/paper/2606.01926/) (2026)
- [The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding](https://mlsanity.com/paper/2603.03305/) (2026)
- [Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models](https://mlsanity.com/paper/2510.07248/) (2025)
- [The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models](https://mlsanity.com/paper/2605.26128/) (2026)
- [Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints](https://mlsanity.com/paper/2606.25605/) (2026)
- [(G)I-DLE: Generative Inference via Distribution-preserving Logit Exclusion with KL Divergence Minimization for Constrained Decoding](https://mlsanity.com/paper/2503.18050/) (2025)
- [From Hallucination to Structure Snowballing: The Alignment Tax of Constrained Decoding in LLM Reflection](https://mlsanity.com/paper/2604.06066/) (2026)
- [Think in English, Answer in Korean: Efficient Adaptation of Multilingual Tool-Using Agents](https://mlsanity.com/paper/2606.31648/) (2026)

## Built on by (most cited first, 4 of 4)

- [Most of the LLM Routing Gap Is Task Type](https://mlsanity.com/paper/2608.23023/) (2026)
- [Protocol Compression Changes Which Party Pays: Bilateral Cost in Cross-Organization LLM Agent Communication](https://mlsanity.com/paper/2609.06129/) (2026)
- [What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects](https://mlsanity.com/paper/2610.05533/) (2026)
- [Structural Automata for Neural Semantic Parsing](https://mlsanity.com/paper/2610.13896/) (2026)

## Abstract (arXiv)

A tool-calling router has to pick the right tool when one applies and decline when none does. Restricting the decoder to a grammar over the tool names is the standard remedy for the first, and on small models it buys a large accuracy gain. Recent work separates the loss caused by asking for a format from the loss caused by enforcing it at decode time. The second is small, which made enforcement look nearly free. The same work declines to extend that to function calling, where a constraint decides which answers exist rather than how one is written. Declining to call anything is the answer it most easily removes, and the one a router can least afford to lose. A grammar decides which tokens may be emitted and where generation stops, and a two-condition design charges both to the restriction. We therefore run three conditions over one prompt: free generation, generation stopped at the first line, and both applied together. We evaluate open-weight models from 0.6B to 4B on the same items in English and Korean, comparing the languages item by item. The two-condition contrast is negative on abstention accuracy in four of six cells with intervals excluding zero, and positive in none, costing -29.5 points at worst. On the smallest model in Korean the stop costs -20.0 points, the restriction returns +19.5, and together they leave -0.5. What the restriction gives back is readable output, not judgment. Of the 698 abstentions it repairs, 545 had no readable answer at all and 0 were correct decisions the scoring rule rejected. On items that do need a tool the contrast is positive throughout, and abstention is reported first because it is the registered measure. Both preregistered claims about language fail: Korean does not lose more of the abstentions it holds without the constraint, and the removed mass does not explain what does.