# InfoFlow: Reinforcing Search Agent Via Reward Density Optimization

arXiv: https://arxiv.org/abs/2510.26575 · HTML: https://mlsanity.com/paper/2510.26575/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 47 of its 58 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 47)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://mlsanity.com/paper/2310.12823/) (2023)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [ZeroSearch: Incentivize the Search Capability of LLMs without Searching](https://mlsanity.com/paper/2505.04588/) (2025)

## Built on by (most cited first, 11 of 11)

- [AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery](https://mlsanity.com/paper/2604.25256/) (2026)
- [SRR-Judge: Step-Level Rating and Refinement for Enhancing Search-Integrated Reasoning in Search Agents](https://mlsanity.com/paper/2602.07773/) (2026)
- [Learning to Retrieve from Agent Trajectories](https://mlsanity.com/paper/2604.04949/) (2026)
- [LAD: Learning Advantage Distribution for Reasoning](https://mlsanity.com/paper/2602.20132/) (2026)
- [CAFE: Self-Improving Search Agents Need Co-Evolving Feedback](https://mlsanity.com/paper/2608.24794/) (2026)
- [D2Plan: Dual-Agent Dynamic Global Planning for Complex Retrieval-Augmented Reasoning](https://mlsanity.com/paper/2601.08282/) (2026)
- [CuSearch: Curriculum Rollout Sampling via Search Depth for Agentic RAG](https://mlsanity.com/paper/2605.11611/) (2026)
- [SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration](https://mlsanity.com/paper/2607.15257/) (2026)
- [Verbal-R3: Verbal Reranker as the Missing Bridge between Retrieval and Reasoning](https://mlsanity.com/paper/2605.01399/) (2026)
- [CAPF: Guiding Search-Agent Rollouts with Credit-Attenuated Privileged Feedback](https://mlsanity.com/paper/2606.01830/) (2026)
- [IterSynth: Rethinking Deep Search Agents via Role-Decoupled Iterative Synthesis](https://mlsanity.com/paper/2609.29444/) (2026)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) is a promising approach for enhancing agentic deep search. However, its application is often hindered by low Reward Density in deep search scenarios, where agents expend significant exploratory costs for infrequent and often null final rewards. In this paper, we formalize this challenge as the Reward Density Optimization problem, which aims to improve the reward obtained per unit of exploration cost. This paper introduce InfoFlow, a systematic framework that tackles this problem from three aspects. 1) Subproblem decomposition: breaking down long-range tasks to assign process rewards, thereby providing denser learning signals. 2) Failure-guided hints: injecting corrective guidance into stalled trajectories to increase the probability of successful outcomes. 3) Dual-agent refinement: employing a dual-agent architecture to offload the cognitive burden of deep exploration. A refiner agent synthesizes the search history, which effectively compresses the researcher's perceived trajectory, thereby reducing exploration cost and increasing the overall reward density. We evaluate InfoFlow on multiple agentic search benchmarks, where it significantly outperforms strong baselines, enabling lightweight LLMs to achieve performance comparable to advanced proprietary LLMs.