mlsanityData to 14 Oct 2026

XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models

arXiv 2411.15100 ↗First posted 22 Nov 2024cs.CL cs.AI cs.PLcited by 75

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

The applications of LLM Agents are becoming increasingly complex and diverse, leading to a high demand for structured outputs that can be parsed into code, structured function calls, and embodied agent commands. These developments bring significant demands for structured generation in LLM inference. Context-free grammar is a flexible approach to enable structured generation via constrained decoding. However, executing context-free grammar requires going through several stack states over all tokens in vocabulary during runtime, bringing non-negligible overhead for structured generation. In this paper, we propose XGrammar, a flexible and efficient structure generation engine for large language models. XGrammar accelerates context-free grammar execution by dividing the vocabulary into context-independent tokens that can be prechecked and context-dependent tokens that need to be interpreted during runtime. We further build transformations to expand the grammar context and reduce the number of context-independent tokens. Additionally, we build an efficient persistent stack to accelerate the context-dependent token checks. Finally, we co-design the grammar engine with LLM inference engine to overlap grammar computation with GPU executions. Evaluation results show that XGrammar can achieve up to 100x speedup over existing solutions. Combined with an LLM inference engine, it can generate near-zero overhead structure generation in end-to-end low-LLM serving.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 17 of its 49 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

17 of 49 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  8. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  9. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  11. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  12. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
5 more
  1. 2021Examining Zero-Shot Vulnerability Repair with Large Language Modelscited by 105
  2. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  3. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  4. 2024Automata-based constraints for language model decodingcited by 51
  5. 2022Validating Large Language Models with ReLMcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 51.

75 papers.

Sort:

By area:

  1. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  2. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  3. 2025Enforcing Temporal Constraints for LLM Agentscited by 34
  4. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 21
  5. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  6. 2025Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherencecited by 16
  7. 2025DINGO: Constrained Inference for Diffusion LLMscited by 16
  8. 2025Social World Modelscited by 16
  9. 2024WebLLM: A High-Performance In-Browser LLM Inference Enginecited by 14
  10. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 10
  11. 2025Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weightscited by 8
  12. 2026Mitigating Bias in Locally Constrained Decoding via Tractable Proposalscited by 8
38 more
  1. 2025Correctness-Guaranteed Code Generation via Constrained Decodingcited by 7
  2. 2026XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMscited by 7
  3. 2026The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decodingcited by 7
  4. 2026In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generationscited by 6
  5. 2025RefineStat: Efficient Exploration for Probabilistic Program Synthesiscited by 5
  6. 2025Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Modelscited by 5
  7. 2026Thinking Before Constraining: A Unified Decoding Framework for Large Language Modelscited by 5
  8. 2026Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automatacited by 4
  9. 2025Metamodel-Guided Model Generation with Layered Constraintscited by 3
  10. 2026CaveAgent: Transforming LLMs into Stateful Runtime Operatorscited by 3
  11. 2026Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generationcited by 3
  12. 2026Typed Decision Models: An Early Evidence Audit and Evaluation Checklistcited by 3
  13. 2025WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decodingcited by 2
  14. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  15. 2025DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correctioncited by 2
  16. 2026NoReGeo: Non-Reasoning Geometry Benchmarkcited by 2
  17. 2026Future Validity is the Missing Statistic: From Impossibility to Phi-Estimation for Grammar-Faithful Speculative Decodingcited by 2
  18. 2026The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputscited by 2
  19. 2026ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflowscited by 2
  20. 2025Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Datacited by 1
  21. 2025You May Speak Freely: Improving the Fine-Grained Visual Recognition Capabilities of Multimodal Large Language Models with Answer Extractioncited by 1
  22. 2026FMBench: Adaptive Large Language Model Output Formattingcited by 1
  23. 2026Designing RNAs with Language Modelscited by 1
  24. 2026Attention Meets Reachability: Structural Equivalence and Efficiency in Grammar-Constrained LLM Decodingcited by 1
  25. 2026Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editingcited by 1
  26. 2026Transcoda: End-to-End Zero-Shot Optical Music Recognition via Data-Centric Synthetic Trainingcited by 1
  27. 2026TruncProof: A Guardrail for LLM-based JSON Generation under Token-Length Constraintscited by 1
  28. 2026The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliabilitycited by 1
  29. 2026Benchmarking System One decision models against trained classifiers and language models for automated decision gatescited by 1
  30. 2025Verbalized Algorithms: Classical Algorithms are All You Need (Mostly)cited by 0
  31. 2025Leveraging Large Language Models to Effectively Generate Visual Data for Canine Musculoskeletal Diagnosescited by 0
  32. 2025Play by the Type Rules: Inferring Constraints for LLM Functions in Declarative Programscited by 0
  33. 2025A Matter of Representation: Towards Graph-Based Abstract Code Generationcited by 0
  34. 2025CAPE: Capability Achievement via Policy Executioncited by 0
  35. 2026The New Compiler Stack: A Survey on the Synergy of LLMs and Compilerscited by 0
  36. 2026Adversarial Query Synthesis via Bayesian Optimizationcited by 0
  37. 2026Training-Free Semantic Multi-Object Tracking with Vision-Language Modelscited by 0
  38. 2026LLM StructCore: Schema-Guided Reasoning Condensation and Deterministic Compilationcited by 0

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Context-Free Grammars for Grammar-Constrained Decoding via Declarative Agentic Programming with Guarantees
  2. 2023Splitwise: Efficient generative LLM inference using phase splitting
  3. 2025Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
  4. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  5. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  6. 2024Adaptable Logical Control for Large Language Models
  7. 2026Write Once, Run Everywhere: The Axon DSL for Shape-Safe and Framework-Agnostic LLM Architectures
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insights