mlsanityData to 14 Oct 2026

Marginal-Contribution Policy Gradients under Filtered Feedback for Multi-Agent LLMs

arXiv 2604.22785 ↗First posted 3 Apr 2026cs.LGcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

We develop a unified treatment of credit assignment for RL training in multi-agent LLM systems. We show that observed reward alone cannot distinguish an agent that determines it from one that never affects it, and that standard shared-reward training performs exact gradient ascent on each agent's private utility rather than system performance. Moreover, we prove no single scalar per agent can consistently account for joint performance once agents interact. We thus develop the unique background-dependent notion of marginal contribution satisfying natural consistency requirements. From it we derive gradient-correct marginal contribution training signals, identify them from filtered feedback, and optimally allocate a budget of exact counterfactual evaluations against learned-signal error. Instantiated in GRPO, our signal improves routed GSM8K accuracy over winner-take-all training at no extra generation cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 21 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

9 of 21 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaboration23 Mar
  2. 2026Exact Is Easier: Credit Assignment for Cooperative LLM Agents6 Mar
  3. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System9 Feb

Other references · 6

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  3. 2019Shapley Q-value: A Local Reward Approach to Solve Global Reward Gamescited by 73
  4. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 68
  5. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  6. 2020Difference Rewards Policy Gradientscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
  2. 2025Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning
  3. 2026COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
  4. 2026Training Small LLMs as Spatial Multi-Agent Policies
  5. 2025Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents
  6. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems
  7. 2026Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
  8. 2026On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse