mlsanityData to 7 Oct 2026

Don't Fine-Tune, Decode: Syntax Error-Free Tool Use via Constrained Decoding

arXiv 2310.07075 ↗First posted 10 Oct 2023cs.CL cs.AIcited by 13

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Instruction-tuned large language models (LLMs) excel at many tasks but often fail to use external tools due to complicated and unfamiliar syntax constraints. While extensive fine-tuning and prompting can mitigate the issue, these approaches are expensive and hard to generalize. Furthermore, because syntax constraints are only learned implicitly during fine-tuning, models still make frequent syntax errors. Motivated by the fact that these constraints can be better satisfied explicitly with constrained decoding, we propose TOOLDEC, a decoding algorithm using finite state machines to force LLMs to follow tool syntax. Our experiments show that TOOLDEC eliminates all syntax errors, achieving significantly better performance on various base models and benchmarks. More surprisingly, when applied to generalist out-of-the-box LLMs such as Mistral-Instruct, TOOLDEC improves its accuracy in tool use from the initial 0% to an impressive 52%, matching the performance of specialized fine-tuned models such as ToolLLM.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 22 of its 32 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

22 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  2. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb
  4. 2022TALM: Tool Augmented Language Models24 May

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  11. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  12. 2023Augmented Language Models: a Surveycited by 398
6 more
  1. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
  2. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
  3. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  4. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  5. 2016Guided Open Vocabulary Image Captioning with Constrained Beam Searchcited by 105
  6. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  2. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  3. 2024Planning and Editing What You Retrieve for Enhanced Tool Learningcited by 21
  4. 2024LLMs in the Imaginarium: Tool Learning through Simulated Trial and Errorcited by 19
  5. 2024Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Accesscited by 9
  6. 2025XML Prompting as Grammar-Constrained Interaction: Fixed-Point Semantics, Convergence Guarantees, and Human-AI Protocolscited by 4
  7. 2025Schema Generation for Large Knowledge Graphs Using Large Language Modelscited by 3
  8. 2026The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliabilitycited by 1
  9. 2026From State to Action: OODA-Tool for Reliable Multi-Turn Tool Usecited by 1
  10. 2026Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decodingcited by 0
  11. 2026Procedural Graphs: Self-Evolving Execution Structures for LLM Agentscited by 0
  12. 2026ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AIcited by 0
1 more
  1. 2026Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learn2Zinc: Fine-tuning Small Language Models for Text-to-Model Translation in MiniZinc
  2. 2026Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap
  3. 2025textttSEM-CTRL: Semantically Controlled Decoding
  4. 2024Grammar-Aligned Decoding
  5. 2025Constrained Decoding of Diffusion LLMs with Context-Free Grammars
  6. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation
  7. 2024Nudging: Inference-time Alignment of LLMs via Guided Decoding
  8. 2025Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?