# RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation

arXiv: https://arxiv.org/abs/2412.11919 · HTML: https://mlsanity.com/paper/2412.11919/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 70 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 30 of 48)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy](https://mlsanity.com/paper/2305.15294/) (2023)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?](https://mlsanity.com/paper/2407.01284/) (2024)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [Generative Representational Instruction Tuning](https://mlsanity.com/paper/2402.09906/) (2024)
- [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://mlsanity.com/paper/2310.05492/) (2023)
- [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022)
- [FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research](https://mlsanity.com/paper/2405.13576/) (2024)
- [End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering](https://mlsanity.com/paper/2106.05346/) (2021)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [A Neural Corpus Indexer for Document Retrieval](https://mlsanity.com/paper/2206.02743/) (2022)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models](https://mlsanity.com/paper/2406.13542/) (2024)

## Built on by (most cited first, 20 of 20)

- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [ZeroSearch: Incentivize the Search Capability of LLMs without Searching](https://mlsanity.com/paper/2505.04588/) (2025)
- [Agentic Reinforced Policy Optimization](https://mlsanity.com/paper/2507.19849/) (2025)
- [DeepAgent: A General Reasoning Agent with Scalable Toolsets](https://mlsanity.com/paper/2510.21618/) (2025)
- [TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2510.07181/) (2025)
- [Hierarchical Document Refinement for Long-context Retrieval-augmented Generation](https://mlsanity.com/paper/2505.10413/) (2025)
- [RAG-R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism](https://mlsanity.com/paper/2507.02962/) (2025)
- [SmartSearch: Process Reward-Guided Query Refinement for Search Agents](https://mlsanity.com/paper/2601.04888/) (2026)
- [Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization](https://mlsanity.com/paper/2511.20718/) (2025)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines](https://mlsanity.com/paper/2505.13538/) (2025)
- [APPO: Agentic Procedural Policy Optimization](https://mlsanity.com/paper/2606.12384/) (2026)
- [Autonomous Knowledge Graph Exploration with Adaptive Breadth-Depth Retrieval](https://mlsanity.com/paper/2601.13969/) (2026)
- [NeocorRAG: Less Irrelevant Information, More Explicit Evidence, and More Effective Recall via Evidence Chains](https://mlsanity.com/paper/2604.27852/) (2026)
- [PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.09287/) (2026)
- [To Know is to Construct: Schema-Constrained Generation for Agent Memory](https://mlsanity.com/paper/2604.20117/) (2026)
- [Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrieval](https://mlsanity.com/paper/2604.23396/) (2026)
- [No=esis: Deterministic-First Retrieval with Two-Tier Context Hydration for Factuality-Critical Queries on Small Local Models](https://mlsanity.com/paper/2609.07663/) (2026)

## Abstract (arXiv)

Large language models (LLMs) exhibit remarkable generative capabilities but often suffer from hallucinations. Retrieval-augmented generation (RAG) offers an effective solution by incorporating external knowledge, but existing methods still face several limitations: additional deployment costs of separate retrievers, redundant input tokens from retrieved text chunks, and the lack of joint optimization of retrieval and generation. To address these issues, we propose RetroLLM, a unified framework that integrates retrieval and generation into a single, cohesive process, enabling LLMs to directly generate fine-grained evidence from the corpus with constrained decoding. Moreover, to mitigate false pruning in the process of constrained evidence generation, we introduce (1) hierarchical FM-Index constraints, which generate corpus-constrained clues to identify a subset of relevant documents before evidence generation, reducing irrelevant decoding space; and (2) a forward-looking constrained decoding strategy, which considers the relevance of future sequences to improve evidence accuracy. Extensive experiments on five open-domain QA datasets demonstrate RetroLLM's superior performance across both in-domain and out-of-domain tasks. The code is available at urlhttps://github.com/sunnynexus/RetroLLM.