# OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Search

arXiv: https://arxiv.org/abs/2604.03675 · HTML: https://mlsanity.com/paper/2604.03675/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 4 Apr 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization](https://mlsanity.com/paper/2505.15107/) (2025)
- [Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning](https://mlsanity.com/paper/2505.14069/) (2025)
- [Bridging the Preference Gap between Retrievers and LLMs](https://mlsanity.com/paper/2401.06954/) (2024)
- [Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.15228/) (2025)
- [Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning](https://mlsanity.com/paper/2505.11277/) (2025)
- [R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning](https://mlsanity.com/paper/2506.04185/) (2025)
- [RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards](https://mlsanity.com/paper/2410.13509/) (2024)
- [DMQR-RAG: Diverse Multi-Query Rewriting for RAG](https://mlsanity.com/paper/2411.13154/) (2024)
- [TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs](https://mlsanity.com/paper/2603.22293/) (2026)
- [A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications](https://mlsanity.com/paper/2510.16724/) (2025)
- [Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training](https://mlsanity.com/paper/2602.22576/) (2026)
- [Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents](https://mlsanity.com/paper/2509.14480/) (2025)

## Built on by (most cited first, 7 of 7)

- [RaMem: Contextual Reinstatement for Long-term Agentic Memory](https://mlsanity.com/paper/2606.22844/) (2026)
- [SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task](https://mlsanity.com/paper/2607.24850/) (2026)
- [Divide and Cooperate: Role-Decomposed Multi-Agent LLM Training with Cross-Agent Learning Signals](https://mlsanity.com/paper/2606.10684/) (2026)
- [SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2610.11345/) (2026)
- [GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs](https://mlsanity.com/paper/2609.12265/) (2026)
- [RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generation](https://mlsanity.com/paper/2609.36652/) (2026)
- [Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems](https://mlsanity.com/paper/2610.01042/) (2026)

## Abstract (arXiv)

Agentic search enables language models to solve knowledge-intensive tasks by adaptively acquiring external evidence over multiple steps. Reinforcement learning with verifiable rewards (RLVR) has emerged as a widely adopted training paradigm for search agents, yet outcome-only rewards are sparse and provide limited credit assignment for intermediate search actions. Existing process-reward methods therefore seek to densify supervision through proxy signals, external evaluators, or likelihood-based information gain. However, proxy rewards can deviate from the final outcome objective, while fixed evaluators can become stale as the search policy evolves, leading to unreliable process supervision. To address these challenges, we propose OASES, an Outcome-Aligned Search-Evaluation Supervision framework for agentic search. OASES derives outcome-aligned process rewards by evaluating how well each intermediate search state supports answering the original question. It further co-trains the search policy and the state evaluator on policy, allowing the evaluator to adapt to evolving search behavior and provide more reliable process rewards. Experiments on five multi-hop QA benchmarks show that OASES consistently outperforms strong RL baselines, with further analyses confirming the benefits of outcome-aligned process rewards and search-evaluation co-training.