mlsanityData to 14 Oct 2026

Guided Decoding and Its Critical Role in Retrieval-Augmented Generation

arXiv 2509.06631 ↗First posted 8 Sep 2025cs.CLcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

The integration of Large Language Models (LLMs) into various applications has driven the need for structured and reliable responses. A key challenge in Retrieval-Augmented Generation (RAG) systems is ensuring that outputs align with expected formats while minimizing hallucinations. This study examines the role of guided decoding in RAG systems, comparing three methods, Outlines, XGrammar, and LM Format Enforcer, across different multi-turn prompting setups (0-turn, 1-turn, and 2-turn). By evaluating success rates, hallucination rates, and output quality, we provide insights into their performance and applicability. Our findings reveal how multi-turn interactions influence guided decoding, uncovering unexpected performance variations that can inform method selection for specific use cases. This work advances the understanding of structured output generation in RAG systems, offering both theoretical insights and practical guidance for LLM deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 7 of its 10 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

7 of 10 references.

References · 7

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Efficient Guided Generation for Large Language Modelscited by 200
  3. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  4. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 80
  5. 2024"We Need Structured Output": Towards User-centered Constraints on Large Language Model Outputcited by 46
  6. 2025StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Followingcited by 29
  7. 2023Guiding Language Models of Code with Global Context using Monitorscited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
  2. 2024A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems
  3. 2025REFRAG: Rethinking RAG based Decoding
  4. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  5. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  6. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  7. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  8. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation