# Marginal-Contribution Policy Gradients under Filtered Feedback for Multi-Agent LLMs

arXiv: https://arxiv.org/abs/2604.22785 · HTML: https://mlsanity.com/paper/2604.22785/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 9 of 9)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Shapley Q-value: A Local Reward Approach to Solve Global Reward Games](https://mlsanity.com/paper/1907.05707/) (2019)
- [Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines](https://mlsanity.com/paper/1803.07246/) (2018)
- [Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2602.01601/) (2026)
- [Counterfactual Credit Policy Optimization for Multi-Agent Collaboration](https://mlsanity.com/paper/2603.21563/) (2026)
- [Exact Is Easier: Credit Assignment for Cooperative LLM Agents](https://mlsanity.com/paper/2603.06859/) (2026)
- [Difference Rewards Policy Gradients](https://mlsanity.com/paper/2012.11258/) (2020)
- [Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System](https://mlsanity.com/paper/2602.08335/) (2026)

## Abstract (arXiv)

We develop a unified treatment of credit assignment for RL training in multi-agent LLM systems. We show that observed reward alone cannot distinguish an agent that determines it from one that never affects it, and that standard shared-reward training performs exact gradient ascent on each agent's private utility rather than system performance. Moreover, we prove no single scalar per agent can consistently account for joint performance once agents interact. We thus develop the unique background-dependent notion of marginal contribution satisfying natural consistency requirements. From it we derive gradient-correct marginal contribution training signals, identify them from filtered feedback, and optimally allocate a budget of exact counterfactual evaluations against learned-signal error. Instantiated in GRPO, our signal improves routed GSM8K accuracy over winner-take-all training at no extra generation cost.