# XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs

arXiv: https://arxiv.org/abs/2601.04426 · HTML: https://mlsanity.com/paper/2601.04426/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 43 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 13 of 13)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models](https://mlsanity.com/paper/2411.15100/) (2024)
- [Grammar Prompting for Domain-Specific Language Generation with Large Language Models](https://mlsanity.com/paper/2305.19234/) (2023)
- [Validating Large Language Models with ReLM](https://mlsanity.com/paper/2211.15458/) (2022)
- [CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions](https://mlsanity.com/paper/2506.01859/) (2025)
- [Efficient Semiring-Weighted Earley Parsing](https://mlsanity.com/paper/2307.02982/) (2023)
- [Directed Graph Hashing](https://mlsanity.com/paper/2002.06653/) (2020)
- [WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decoding](https://mlsanity.com/paper/2507.16768/) (2025)

## Built on by (most cited first, 7 of 7)

- [VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?](https://mlsanity.com/paper/2605.06068/) (2026)
- [Future Validity is the Missing Statistic: From Impossibility to Phi-Estimation for Grammar-Faithful Speculative Decoding](https://mlsanity.com/paper/2605.07698/) (2026)
- [Accelerating Constrained Decoding with Token Space Compression](https://mlsanity.com/paper/2605.29986/) (2026)
- [Neurosymbolic Embodied Agents](https://mlsanity.com/paper/2608.16794/) (2026)
- [Parser Transitions Already Reveal Structural Responsibility: KV Persistence for Structured Generation](https://mlsanity.com/paper/2608.28276/) (2026)
- [Provably Tractable NFA-Constrained Language Generation via HMMs](https://mlsanity.com/paper/2609.40185/) (2026)
- [SchemaWeaver: Accelerating On-Device Mobile Agentic AI via Schema-Guided Parallel LLM Decoding](https://mlsanity.com/paper/2610.17436/) (2026)

## Abstract (arXiv)

Modern LLM agents increasingly rely on dynamic structured generation, such as tool calling and response protocols. Unlike traditional structured generation with static structures, these workloads vary both across requests and within a request, posing new challenges to existing engines. We present XGrammar-2, a structured generation engine for dynamic agentic workloads. Our design is based on two key ideas: first-class support for tag-triggered structure switching, and fine-grained reuse across requests with different output structures. Concretely, XGrammar-2 introduces TagDispatch for dynamic structural dispatching and Cross-Grammar Cache for substructure-level cache reuse across grammars. It further improves efficiency with an Earley-based adaptive token mask cache, just-in-time compilation, and repetition state compression. Experiments show that XGrammar-2 achieves over 6x faster compilation than prior structured generation engines, and incurs near-zero end-to-end overhead in modern LLM serving systems.