# Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning

arXiv: https://arxiv.org/abs/2305.13971 · HTML: https://mlsanity.com/paper/2305.13971/

## Key facts

- Cited by 197 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [Recurrent Neural Network Grammars](https://mlsanity.com/paper/1602.07776/) (2016)
- [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://mlsanity.com/paper/1612.00576/) (2016)
- [Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction](https://mlsanity.com/paper/2303.04132/) (2023)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Deep Joint Entity Disambiguation with Local Neural Attention](https://mlsanity.com/paper/1704.04920/) (2017)
- [ReFinED: An Efficient Zero-shot-capable Approach to End-to-End Entity Linking](https://mlsanity.com/paper/2207.04108/) (2022)
- [Grammar as a Foreign Language](https://mlsanity.com/paper/1412.7449/) (2014)
- [Improving Entity Linking by Modeling Latent Relations between Mentions](https://mlsanity.com/paper/1804.10637/) (2018)
- [GenIE: Generative Information Extraction](https://mlsanity.com/paper/2112.08340/) (2021)
- [Fast and Accurate Neural CRF Constituency Parsing](https://mlsanity.com/paper/2008.03736/) (2020)
- [Autoregressive Structured Prediction with Language Models](https://mlsanity.com/paper/2210.14698/) (2022)
- [Augmented Natural Language for Generative Sequence Labeling](https://mlsanity.com/paper/2009.13272/) (2020)

## Built on by (most cited first, 30 of 197)

- [Securing AI Agents with Information-Flow Control](https://mlsanity.com/paper/2505.23643/) (2025)
- [Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation](https://mlsanity.com/paper/2403.06988/) (2024)
- [GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer](https://mlsanity.com/paper/2311.08526/) (2023)
- [Perception Tokens Enhance Visual Reasoning in Multimodal Language Models](https://mlsanity.com/paper/2412.03548/) (2024)
- [JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models](https://mlsanity.com/paper/2501.10868/) (2025)
- [SynCode: LLM Generation with Grammar Augmentation](https://mlsanity.com/paper/2403.01632/) (2024)
- [Grammar-Aligned Decoding](https://mlsanity.com/paper/2405.21047/) (2024)
- [Automata-based constraints for language model decoding](https://mlsanity.com/paper/2407.08103/) (2024)
- [Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo](https://mlsanity.com/paper/2504.13139/) (2025)
- [KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection](https://mlsanity.com/paper/2310.09044/) (2023)
- [Flexible and Efficient Grammar-Constrained Decoding](https://mlsanity.com/paper/2502.05111/) (2025)
- [Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents](https://mlsanity.com/paper/2402.00798/) (2024)
- [From Language Models over Tokens to Language Models over Characters](https://mlsanity.com/paper/2412.03719/) (2024)
- [IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking](https://mlsanity.com/paper/2410.07295/) (2024)
- [Learning to Generate Structured Output with Schema Reinforcement Learning](https://mlsanity.com/paper/2502.18878/) (2025)
- [Octopus: On-device language model for function calling of software APIs](https://mlsanity.com/paper/2404.01549/) (2024)
- [StructuredRAG: JSON Response Formatting with Large Language Models](https://mlsanity.com/paper/2408.11061/) (2024)
- [To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning](https://mlsanity.com/paper/2410.16456/) (2024)
- [Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling](https://mlsanity.com/paper/2504.05410/) (2025)
- [Constrained Discrete Diffusion](https://mlsanity.com/paper/2503.09790/) (2025)
- [Unveiling the Potential of Diffusion Large Language Model in Controllable Generation](https://mlsanity.com/paper/2507.04504/) (2025)
- [Constrained Sampling for Language Models Should Be Easy: An MCMC Perspective](https://mlsanity.com/paper/2506.05754/) (2025)
- [Don't Fine-Tune, Decode: Syntax Error-Free Tool Use via Constrained Decoding](https://mlsanity.com/paper/2310.07075/) (2023)
- [Group of Skills: Group-Structured Skill Retrieval for Agent Skill Libraries](https://mlsanity.com/paper/2605.06978/) (2026)
- [Comparing GPT-4 and Open-Source Language Models in Misinformation Mitigation](https://mlsanity.com/paper/2401.06920/) (2024)
- [Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access](https://mlsanity.com/paper/2401.09967/) (2024)
- [Grammar-based Game Description Generation using Large Language Models](https://mlsanity.com/paper/2407.17404/) (2024)
- [A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models](https://mlsanity.com/paper/2410.04027/) (2024)
- [Shh, don't say that! Domain Certification in LLMs](https://mlsanity.com/paper/2502.19320/) (2025)
- [GraphRAFT: Retrieval Augmented Fine-Tuning for Knowledge Graphs on Graph Databases](https://mlsanity.com/paper/2504.05478/) (2025)

## Abstract (arXiv)

Despite their impressive performance, large language models (LMs) still struggle with reliably generating complex output structures when not finetuned to follow the required output format exactly. To address this issue, grammar-constrained decoding (GCD) can be used to control the generation of LMs, guaranteeing that the output follows a given structure. Most existing GCD methods are, however, limited to specific tasks, such as parsing or code generation. In this work, we demonstrate that formal grammars can describe the output space for a much wider range of tasks and argue that GCD can serve as a unified framework for structured NLP tasks in general. For increased flexibility, we introduce input-dependent grammars, which allow the grammar to depend on the input and thus enable the generation of different output structures for different inputs. We then empirically demonstrate the power and flexibility of GCD-enhanced LMs on (1) information extraction, (2) entity disambiguation, and (3) constituency parsing. Our results indicate that grammar-constrained LMs substantially outperform unconstrained LMs or even beat task-specific finetuned models. Grammar constraints thus hold great promise for harnessing off-the-shelf LMs for a wide range of structured NLP tasks, especially where training data is scarce or finetuning is expensive. Code and data: https://github.com/epfl-dlab/GCD.