# Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval

arXiv: https://arxiv.org/abs/2410.23214 · HTML: https://mlsanity.com/paper/2410.23214/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020)
- [Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents](https://mlsanity.com/paper/2403.02502/) (2024)
- [Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2311.09210/) (2023)
- [Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval](https://mlsanity.com/paper/2009.12756/) (2020)
- [Query2doc: Query Expansion with Large Language Models](https://mlsanity.com/paper/2303.07678/) (2023)
- [Rethinking with Retrieval: Faithful Large Language Model Inference](https://mlsanity.com/paper/2301.00303/) (2022)
- [Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms](https://mlsanity.com/paper/2406.02900/) (2024)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [Internet-augmented language models through few-shot prompting for open-domain question answering](https://mlsanity.com/paper/2203.05115/) (2022)
- [Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval](https://mlsanity.com/paper/2101.00436/) (2021)
- [Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together](https://mlsanity.com/paper/2407.10930/) (2024)

## Built on by (most cited first, 17 of 17)

- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges](https://mlsanity.com/paper/2508.05668/) (2025)
- [Synergizing RAG and Reasoning: A Systematic Review](https://mlsanity.com/paper/2504.15909/) (2025)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Learning Next Action Predictors from Human-Computer Interaction](https://mlsanity.com/paper/2603.05923/) (2026)
- [Improving Context Fidelity via Native Retrieval-Augmented Reasoning](https://mlsanity.com/paper/2509.13683/) (2025)
- [FrugalRAG: Less is More in RL Finetuning for Multi-Hop Question Answering](https://mlsanity.com/paper/2507.07634/) (2025)
- [Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models](https://mlsanity.com/paper/2603.13985/) (2026)
- [S3-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data](https://mlsanity.com/paper/2605.01248/) (2026)
- [Beyond Text: Characterizing Domain Expert Needs in Document Research](https://mlsanity.com/paper/2504.12495/) (2025)
- [Lucy: edgerunning agentic web search on mobile with machine generated task vectors](https://mlsanity.com/paper/2508.00360/) (2025)
- [Foundational Design Principles and Patterns for Building Robust and Adaptive GenAI-Native Systems](https://mlsanity.com/paper/2508.15411/) (2025)
- [QueStER: Query Specification for Generative keyword-based Retrieval](https://mlsanity.com/paper/2511.05301/) (2025)
- [MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.04969/) (2026)
- [Document Optimization for Black-Box Retrieval via Reinforcement Learning](https://mlsanity.com/paper/2604.05087/) (2026)
- [CoHyDE: Iterative Co-Training of LLM Rewriter & Dense Encoder for Tool Retrieval](https://mlsanity.com/paper/2605.29271/) (2026)

## Abstract (arXiv)

The hallucinations of large language models (LLMs) are increasingly mitigated by allowing LLMs to search for information and to ground their answers in real sources. Unfortunately, LLMs often struggle with posing the right search queries, especially when dealing with complex or otherwise indirect topics. Observing that LLMs can learn to search for relevant facts by textittrying different queries and learning to up-weight queries that successfully produce relevant results, we introduce underlineLearning to underlineRetrieve by underlineTrying (LeReT), a reinforcement learning framework that explores search queries and uses preference-based optimization to improve their quality. LeReT can improve the absolute retrieval accuracy by up to 29% and the downstream generator evaluations by 17%. The simplicity and flexibility of LeReT allows it to be applied to arbitrary off-the-shelf retrievers and makes it a promising technique for improving general LLM pipelines. Project website: http://sherylhsu.com/LeReT/.