mlsanityData to 14 Oct 2026

Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization

arXiv 2604.14267 ↗First posted 15 Apr 2026cs.LG cs.AIcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Search agents extend Large Language Models (LLMs) beyond static parametric knowledge by enabling access to up-to-date and long-tail information unavailable during pretraining. While reinforcement learning has been widely adopted for training such agents, existing approaches face key limitations: process supervision often suffers from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. To bridge this gap, we propose Contribution-Weighted GRPO (CW-GRPO), a framework that integrates process supervision into group relative policy optimization. Instead of directly optimizing process rewards, CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores. These scores are used to rescale outcome-based advantages along the trajectory, enabling fine-grained credit assignment without sacrificing optimization stability. Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B, leading to more effective search behaviors. Additional analysis reveals that successful trajectories exhibit concentrated contributions in specific rounds, providing empirical insight into search agent tasks.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 48 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

40 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment17 May
  2. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  3. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization24 Dec
  4. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 35

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  12. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
23 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  3. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  4. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  5. 2023Active Retrieval Augmented Generationcited by 516
  6. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  8. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  9. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  10. 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 180
  11. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  12. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  13. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  14. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  15. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrieverscited by 84
  16. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  17. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  18. 2025Deep Research: A Survey of Autonomous Research Agentscited by 66
  19. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  20. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 30
  21. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  22. 2025R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learningcited by 20
  23. 2024Multi-Reranker: Maximizing performance of retrieval-augmented generation in the FinanceRAG challengecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Trainingcited by 6
  2. 2026TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agentscited by 3
  3. 2026Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learningcited by 2
  4. 2026EviBack: Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoffcited by 0
  5. 2026SIPO: Selective-Inference Policy Optimization for Tree-Structured Agentic RLcited by 0
  6. 2026Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learningcited by 0
  7. 2026Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learningcited by 0
  8. 2026Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agentscited by 0
  9. 2026C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Training-Free Group Relative Policy Optimization
  2. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  3. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  4. 2025Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents
  5. 2026GRPO Training Dynamics for Small Language Models
  6. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  7. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
  8. 2026iGRPO: Self-Feedback-Driven LLM Reasoning