# Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System

arXiv: https://arxiv.org/abs/2602.08335 · HTML: https://mlsanity.com/paper/2602.08335/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [Agentic Reinforced Policy Optimization](https://mlsanity.com/paper/2507.19849/) (2025)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [Algorithms to estimate Shapley value feature attributions](https://mlsanity.com/paper/2207.07605/) (2022)
- [Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification](https://mlsanity.com/paper/2407.20859/) (2024)
- [Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL](https://mlsanity.com/paper/2508.13167/) (2025)
- [MALT: Improving Reasoning with Multi-Agent LLM Training](https://mlsanity.com/paper/2412.01928/) (2024)
- [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023)
- [DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents](https://mlsanity.com/paper/2311.09805/) (2023)
- [SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models](https://mlsanity.com/paper/2509.23863/) (2025)
- [Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO](https://mlsanity.com/paper/2511.13288/) (2025)
- [Multi-Agent Tool-Integrated Policy Optimization](https://mlsanity.com/paper/2510.04678/) (2025)
- [AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play](https://mlsanity.com/paper/2509.24193/) (2025)
- [FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging](https://mlsanity.com/paper/2508.04625/) (2025)
- [Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey](https://mlsanity.com/paper/2505.03418/) (2025)
- [One Step is Enough: Multi-Agent Reinforcement Learning based on One-Step Policy Optimization for Order Dispatch on Ride-Sharing Platforms](https://mlsanity.com/paper/2507.15351/) (2025)

## Built on by (most cited first, 7 of 7)

- [From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2604.09459/) (2026)
- [CARD: Towards Conditional Design of Multi-agent Topological Structures](https://mlsanity.com/paper/2603.01089/) (2026)
- [Exact Is Easier: Credit Assignment for Cooperative LLM Agents](https://mlsanity.com/paper/2603.06859/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces](https://mlsanity.com/paper/2605.02801/) (2026)
- [Marginal-Contribution Policy Gradients under Filtered Feedback for Multi-Agent LLMs](https://mlsanity.com/paper/2604.22785/) (2026)
- [Multi-Agent System Search via Active Substructure-aware Policy Optimization](https://mlsanity.com/paper/2609.32430/) (2026)

## Abstract (arXiv)

Integrating Large Language Models (LLMs) with external tools via multi-agent systems offers a promising new paradigm for decomposing and solving complex problems. However, training these systems remains notoriously difficult due to the credit assignment challenge, as it is often unclear which specific functional agent is responsible for the success or failure of decision trajectories. Existing methods typically rely on sparse or globally broadcast rewards, failing to capture individual contributions and leading to inefficient reinforcement learning. To address these limitations, we introduce the Shapley-based Hierarchical Attribution for Reinforcement Policy (SHARP), a novel framework for optimizing multi-agent reinforcement learning via precise credit attribution. SHARP effectively stabilizes training by normalizing agent-specific advantages across trajectory groups, primarily through a decomposed reward mechanism comprising a global broadcast-accuracy reward, a Shapley-based marginal-credit reward for each agent, and a tool-process reward to improve execution efficiency. Extensive experiments across various real-world benchmarks demonstrate that SHARP significantly outperforms recent state-of-the-art baselines, achieving average match improvements of 23.66% and 14.05% over single-agent and multi-agent approaches, respectively.