# Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents

arXiv: https://arxiv.org/abs/2610.10179 · HTML: https://mlsanity.com/paper/2610.10179/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 9 of 9)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents](https://mlsanity.com/paper/2505.15117/) (2025)
- [R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.23794/) (2025)
- [STAMP: Provenance-Guided Credit Assignment for Deep Search Agents](https://mlsanity.com/paper/2607.11172/) (2026)
- [Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling](https://mlsanity.com/paper/2605.29697/) (2026)
- [BiCAA: Bidirectional Credit Assignment for Search-Augmented Agent](https://mlsanity.com/paper/2608.01321/) (2026)

## Abstract (arXiv)

Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.