# CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic

arXiv: https://arxiv.org/abs/2511.12159 · HTML: https://mlsanity.com/paper/2511.12159/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)
- [SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning](https://mlsanity.com/paper/2509.02479/) (2025)
- [SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks](https://mlsanity.com/paper/2503.15478/) (2025)
- [Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL](https://mlsanity.com/paper/2508.13167/) (2025)
- [Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment](https://mlsanity.com/paper/2505.11821/) (2025)
- [SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution](https://mlsanity.com/paper/2505.20732/) (2025)
- [Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning](https://mlsanity.com/paper/2505.14069/) (2025)
- [Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning](https://mlsanity.com/paper/2505.00024/) (2025)
- [Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL](https://mlsanity.com/paper/2505.10832/) (2025)
- [Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization](https://mlsanity.com/paper/2412.18279/) (2024)
- [LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization](https://mlsanity.com/paper/2505.17447/) (2025)
- [Empowering Multi-Robot Cooperation via Sequential World Models](https://mlsanity.com/paper/2509.13095/) (2025)

## Built on by (most cited first, 30 of 30)

- [Dynamic Dual-Granularity Skill Bank for Agentic RL](https://mlsanity.com/paper/2603.28716/) (2026)
- [From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2604.09459/) (2026)
- [Are Full Rollouts Necessary for On-Policy Distillation?](https://mlsanity.com/paper/2605.31490/) (2026)
- [On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement](https://mlsanity.com/paper/2512.04220/) (2025)
- [Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agents](https://mlsanity.com/paper/2603.12634/) (2026)
- [MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching](https://mlsanity.com/paper/2601.10712/) (2026)
- [Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization](https://mlsanity.com/paper/2604.14267/) (2026)
- [Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning](https://mlsanity.com/paper/2605.22511/) (2026)
- [SE-Search: Self-Evolving Search Agent via Memory and Dense Reward](https://mlsanity.com/paper/2603.03293/) (2026)
- [π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data](https://mlsanity.com/paper/2604.14054/) (2026)
- [PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment](https://mlsanity.com/paper/2606.09348/) (2026)
- [TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization](https://mlsanity.com/paper/2601.22776/) (2026)
- [BranPO: Scalable Contrastive Branch Sampling for Long-Horizon Agentic Reinforcement Learning](https://mlsanity.com/paper/2602.03719/) (2026)
- [Improving Search Agent with One Line of Code](https://mlsanity.com/paper/2603.10069/) (2026)
- [ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget](https://mlsanity.com/paper/2604.01195/) (2026)
- [Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents](https://mlsanity.com/paper/2608.12764/) (2026)
- [CAFE: Self-Improving Search Agents Need Co-Evolving Feedback](https://mlsanity.com/paper/2608.24794/) (2026)
- [Skip-Connected Policy Optimization for Implicit Advantage](https://mlsanity.com/paper/2604.08690/) (2026)
- [Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents](https://mlsanity.com/paper/2604.18235/) (2026)
- [Plan Before Search: Search Agents Need Plan](https://mlsanity.com/paper/2605.28354/) (2026)
- [EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agents](https://mlsanity.com/paper/2608.01359/) (2026)
- [TIDE-Bench: Task-Aware and Diagnostic Evaluation of Tool-Integrated Reasoning](https://mlsanity.com/paper/2605.09544/) (2026)
- [Scaling Retrieval-Augmented Reasoning with Parallel Search and Explicit Merging](https://mlsanity.com/paper/2605.13534/) (2026)
- [Peak-Then-Collapse and the Four Interface Channels of Knowledge-Graph Tool Use](https://mlsanity.com/paper/2605.26037/) (2026)
- [Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM](https://mlsanity.com/paper/2607.18915/) (2026)
- [Harness-G: A Graph-Structured Harness for Search Agents](https://mlsanity.com/paper/2607.27652/) (2026)
- [Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards](https://mlsanity.com/paper/2608.07531/) (2026)
- [CoBRA: Learning Tool-Use Boundaries via Counterfactual Margins](https://mlsanity.com/paper/2609.00967/) (2026)
- [Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.36178/) (2026)
- [Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents](https://mlsanity.com/paper/2610.16027/) (2026)

## Abstract (arXiv)

Tool-Integrated Reasoning (TIR) with search engines enables large language models to iteratively retrieve up-to-date external knowledge, enhancing adaptability and generalization in complex question-answering tasks. However, existing search agent pipelines typically depend on reinforcement learning based optimization, which often suffers from sparse outcome rewards, leading to inefficient exploration and unstable training. We introduce CriticSearch, a fine-grained credit-assignment framework that supplies dense, turn-level feedback via a retrospective critic mechanism. During training, a frozen, asymmetric critique LLM retrospectively evaluates each turn using privileged information from the full trajectory and gold answers, converting these assessments into stable, dense rewards that guide policy improvement. Experimental results across diverse multi-hop reasoning benchmarks demonstrate that CriticSearch consistently outperforms existing baselines, achieving faster convergence, improved training stability, and higher performance.