# Learning to Retrieve from Agent Trajectories

arXiv: https://arxiv.org/abs/2604.04949 · HTML: https://mlsanity.com/paper/2604.04949/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 26 of 26)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://mlsanity.com/paper/2508.06600/) (2025)
- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents](https://mlsanity.com/paper/2509.06501/) (2025)
- [Evaluating Retrieval Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.13781/) (2024)
- [Bridging the Preference Gap between Retrievers and LLMs](https://mlsanity.com/paper/2401.06954/) (2024)
- [Tongyi DeepResearch Technical Report](https://mlsanity.com/paper/2510.24701/) (2025)
- [A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges](https://mlsanity.com/paper/2508.05668/) (2025)
- [A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications](https://mlsanity.com/paper/2506.12594/) (2025)
- [Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximization](https://mlsanity.com/paper/2405.02816/) (2024)
- [Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.18676/) (2024)
- [Deep Research: A Systematic Survey](https://mlsanity.com/paper/2512.02038/) (2025)
- [Are Large Language Models Good at Utility Judgments?](https://mlsanity.com/paper/2403.19216/) (2024)
- [Open Data Synthesis For Deep Research](https://mlsanity.com/paper/2509.00375/) (2025)
- [Advancing the Search Frontier with AI Agents](https://mlsanity.com/paper/2311.01235/) (2023)
- [Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers](https://mlsanity.com/paper/2505.20128/) (2025)
- [InfoFlow: Reinforcing Search Agent Via Reward Density Optimization](https://mlsanity.com/paper/2510.26575/) (2025)
- [NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search](https://mlsanity.com/paper/2505.14680/) (2025)

## Built on by (most cited first, 3 of 3)

- [Masking Stale Observations Helps Search Agents -- Until It Doesn't: A Regime Map and Its Mechanism](https://mlsanity.com/paper/2606.00408/) (2026)
- [CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents](https://mlsanity.com/paper/2608.05886/) (2026)
- [ITER: Interaction-Aware Retrieval for Agentic Search](https://mlsanity.com/paper/2608.27912/) (2026)

## Abstract (arXiv)

Information retrieval (IR) systems have traditionally been designed and trained for human users, with learning-to-rank methods relying heavily on large-scale human interaction logs such as clicks and dwell time. With the rapid emergence of large language model (LLM) powered search agents, however, retrieval is increasingly consumed by agents rather than human beings, and is embedded as a core component within multi-turn reasoning and action loops. In this setting, retrieval models trained under human-centric assumptions exhibit a fundamental mismatch with the way agents issue queries and consume results. In this work, we argue that retrieval models for agentic search should be trained directly from agent interaction data. We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions. Through a systematic analysis of search agent trajectories, we identify key behavioral signals that reveal document utility, including browsing actions, unbrowsed rejections, and post-browse reasoning traces. Guided by these insights, we propose LRAT, a simple yet effective framework that mines high-quality retrieval supervision from agent trajectories and incorporates relevance intensity through weighted optimization. Extensive experiments on both in-domain and out-of-domain deep research benchmarks demonstrate that retrievers trained with LRAT consistently improve evidence recall, end-to-end task success, and execution efficiency across diverse agent architectures and scales. Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.