# JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

arXiv: https://arxiv.org/abs/2501.10868 · HTML: https://mlsanity.com/paper/2501.10868/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 18 Jan 2025.

## Builds on (most cited first, 17 of 17)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Efficient Guided Generation for Large Language Models](https://mlsanity.com/paper/2307.09702/) (2023)
- [Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning](https://mlsanity.com/paper/2305.13971/) (2023)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [Prompting Is Programming: A Query Language for Large Language Models](https://mlsanity.com/paper/2212.06094/) (2022)
- [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://mlsanity.com/paper/2408.02442/) (2024)
- [Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering](https://mlsanity.com/paper/2303.05352/) (2023)
- [XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models](https://mlsanity.com/paper/2411.15100/) (2024)
- [COLLIE: Systematic Construction of Constrained Text Generation Tasks](https://mlsanity.com/paper/2307.08689/) (2023)
- [Grammar Prompting for Domain-Specific Language Generation with Large Language Models](https://mlsanity.com/paper/2305.19234/) (2023)
- [Validating Large Language Models with ReLM](https://mlsanity.com/paper/2211.15458/) (2022)
- [BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing](https://mlsanity.com/paper/2206.10668/) (2022)
- [Byte BPE Tokenization as an Inverse string Homomorphism](https://mlsanity.com/paper/2412.03160/) (2024)
- [Witness Generation for JSON Schema](https://mlsanity.com/paper/2202.12849/) (2022)

## Built on by (most cited first, 30 of 71)

- [Flexible and Efficient Grammar-Constrained Decoding](https://mlsanity.com/paper/2502.05111/) (2025)
- [Learning to Generate Structured Output with Schema Reinforcement Learning](https://mlsanity.com/paper/2502.18878/) (2025)
- [BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP](https://mlsanity.com/paper/2506.10896/) (2025)
- [SLOT: Structuring the Output of Large Language Models](https://mlsanity.com/paper/2505.04016/) (2025)
- [Constrained Sampling for Language Models Should Be Easy: An MCMC Perspective](https://mlsanity.com/paper/2506.05754/) (2025)
- [Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study](https://mlsanity.com/paper/2505.15389/) (2025)
- [Mitigating Bias in Locally Constrained Decoding via Tractable Proposals](https://mlsanity.com/paper/2606.01926/) (2026)
- [SO-Bench: A Structural Output Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2511.21750/) (2025)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [LLMStructBench: Benchmarking Large Language Model Structured Data Extraction](https://mlsanity.com/paper/2602.14743/) (2026)
- [When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output](https://mlsanity.com/paper/2503.24191/) (2025)
- [Semantic Probabilistic Control of Language Models](https://mlsanity.com/paper/2505.01954/) (2025)
- [VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents](https://mlsanity.com/paper/2603.15118/) (2026)
- [VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?](https://mlsanity.com/paper/2605.06068/) (2026)
- [Constrained Adaptive Rejection Sampling](https://mlsanity.com/paper/2510.01902/) (2025)
- [The Format Tax](https://mlsanity.com/paper/2604.03616/) (2026)
- [PatchBoard: Schema-Grounded State Mutation for Reliable and Auditable LLM Multi-Agent Collaboration](https://mlsanity.com/paper/2605.29313/) (2026)
- [Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Scheduling](https://mlsanity.com/paper/2508.03611/) (2025)
- [Retrieving Climate Change Disinformation by Narrative](https://mlsanity.com/paper/2603.22015/) (2026)
- [The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs](https://mlsanity.com/paper/2605.08737/) (2026)
- [Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety](https://mlsanity.com/paper/2605.12729/) (2026)
- [Your Prompt Is Not the Only Prompt: How Much Do LLMs Weight Structured-Output Schema Descriptions?](https://mlsanity.com/paper/2608.08254/) (2026)
- [Attention Meets Reachability: Structural Equivalence and Efficiency in Grammar-Constrained LLM Decoding](https://mlsanity.com/paper/2603.05540/) (2026)
- [SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks](https://mlsanity.com/paper/2603.10002/) (2026)
- [Teaching an Agent to Sketch One Part at a Time](https://mlsanity.com/paper/2603.19500/) (2026)
- [SEAR: Schema-Based Evaluation and Routing for LLM Gateways](https://mlsanity.com/paper/2603.26728/) (2026)
- [Diagnosing CFG Interpretation in LLMs](https://mlsanity.com/paper/2604.20811/) (2026)
- [When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models](https://mlsanity.com/paper/2605.02363/) (2026)
- [DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models](https://mlsanity.com/paper/2605.16342/) (2026)
- [It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers](https://mlsanity.com/paper/2605.26731/) (2026)

## Abstract (arXiv)

Reliably generating structured outputs has become a critical capability for modern language model (LM) applications. Constrained decoding has emerged as the dominant technology across sectors for enforcing structured outputs during generation. Despite its growing adoption, little has been done with the systematic evaluation of the behaviors and performance of constrained decoding. Constrained decoding frameworks have standardized around JSON Schema as a structured data format, with most uses guaranteeing constraint compliance given a schema. However, there is poor understanding of the effectiveness of the methods in practice. We present an evaluation framework to assess constrained decoding approaches across three critical dimensions: efficiency in generating constraint-compliant outputs, coverage of diverse constraint types, and quality of the generated outputs. To facilitate this evaluation, we introduce JSONSchemaBench, a benchmark for constrained decoding comprising 10K real-world JSON schemas that encompass a wide range of constraints with varying complexity. We pair the benchmark with the existing official JSON Schema Test Suite and evaluate six state-of-the-art constrained decoding frameworks, including Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini. Through extensive experiments, we gain insights into the capabilities and limitations of constrained decoding on structured generation with real-world JSON schemas. Our work provides actionable insights for improving constrained decoding frameworks and structured generation tasks, setting a new standard for evaluating constrained decoding and structured generation. We release JSONSchemaBench at https://github.com/guidance-ai/jsonschemabench