mlsanityData to 14 Oct 2026

CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic

arXiv 2511.12159 ↗First posted 15 Nov 2025cs.CLcited by 30

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Tool-Integrated Reasoning (TIR) with search engines enables large language models to iteratively retrieve up-to-date external knowledge, enhancing adaptability and generalization in complex question-answering tasks. However, existing search agent pipelines typically depend on reinforcement learning based optimization, which often suffers from sparse outcome rewards, leading to inefficient exploration and unstable training. We introduce CriticSearch, a fine-grained credit-assignment framework that supplies dense, turn-level feedback via a retrospective critic mechanism. During training, a frozen, asymmetric critique LLM retrospectively evaluates each turn using privileged information from the full trajectory and gold answers, converting these assessments into stable, dense rewards that guide policy improvement. Experimental results across diverse multi-hop reasoning benchmarks demonstrate that CriticSearch consistently outperforms existing baselines, achieving faster convergence, improved training stability, and higher performance.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 45 references that are in the map. First posted on arXiv 15 Nov 2025.

Builds on

31 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL11 Aug
  2. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning20 May

Other references · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  12. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
17 more
  1. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  2. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  3. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  4. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  5. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  6. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  7. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  8. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
  9. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
  10. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 79
  11. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 73
  12. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 68
  13. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 62
  14. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimizationcited by 17
  15. 2025LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridizationcited by 10
  16. 2025Empowering Multi-Robot Cooperation via Sequential World Modelscited by 9
  17. 2025Improving Value Estimation Critically Enhances Vanilla Policy Gradientcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 29.

30 papers.

Sort:

By area:

  1. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 34
  2. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 23
  3. 2026Are Full Rollouts Necessary for On-Policy Distillation?cited by 17
  4. 2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacementcited by 15
  5. 2026Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agentscited by 11
  6. 2026MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matchingcited by 10
  7. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimizationcited by 9
  8. 2026Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoningcited by 9
  9. 2026SE-Search: Self-Evolving Search Agent via Memory and Dense Rewardcited by 8
  10. 2026π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Datacited by 7
  11. 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
  12. 2026TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimizationcited by 5
18 more
  1. 2026BranPO: Scalable Contrastive Branch Sampling for Long-Horizon Agentic Reinforcement Learningcited by 3
  2. 2026Improving Search Agent with One Line of Codecited by 2
  3. 2026ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budgetcited by 2
  4. 2026Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agentscited by 2
  5. 2026CAFE: Self-Improving Search Agents Need Co-Evolving Feedbackcited by 2
  6. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  7. 2026Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agentscited by 1
  8. 2026Plan Before Search: Search Agents Need Plancited by 1
  9. 2026EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agentscited by 1
  10. 2026TIDE-Bench: Task-Aware and Diagnostic Evaluation of Tool-Integrated Reasoningcited by 0
  11. 2026Scaling Retrieval-Augmented Reasoning with Parallel Search and Explicit Mergingcited by 0
  12. 2026Peak-Then-Collapse and the Four Interface Channels of Knowledge-Graph Tool Usecited by 0
  13. 2026Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLMcited by 0
  14. 2026Harness-G: A Graph-Structured Harness for Search Agentscited by 0
  15. 2026Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewardscited by 0
  16. 2026CoBRA: Learning Tool-Use Boundaries via Counterfactual Marginscited by 0
  17. 2026Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learningcited by 0
  18. 2026Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation
  2. 2026RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents
  3. 2026ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment
  4. 2025RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generation
  5. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
  6. 2026Critic-R: Improving Agentic Search using Instruction-tuned Retrievers with Natural Language Introspective Feedback
  7. 2026Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration
  8. 2026Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents