# Q-value Path Decomposition for Deep Multiagent Reinforcement Learning

arXiv: https://arxiv.org/abs/2002.03950 · HTML: https://mlsanity.com/paper/2002.03950/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 24 of 24)

- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00285/) (2021)
- [Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach](https://mlsanity.com/paper/2108.02731/) (2021)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on states](https://mlsanity.com/paper/2205.07229/) (2022)
- [Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.15612/) (2022)
- [Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.04612/) (2022)
- [Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.03902/) (2022)
- [Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.06042/) (2021)
- [Concentration Network for Reinforcement Learning of Large-Scale Multi-Agent Systems](https://mlsanity.com/paper/2203.06416/) (2022)
- [Grounded Answers for Multi-agent Decision-making Problem through Generative World Model](https://mlsanity.com/paper/2410.02664/) (2024)
- [Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2211.12075/) (2022)
- [Efficiently Quantifying Individual Agent Importance in Cooperative MARL](https://mlsanity.com/paper/2312.08466/) (2023)
- [S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?](https://mlsanity.com/paper/2206.11054/) (2022)
- [Cooperative Multi-Agent Reinforcement Learning for Inventory Management](https://mlsanity.com/paper/2304.08769/) (2023)
- [Learning Heterogeneous Agent Cooperation via Multiagent League Training](https://mlsanity.com/paper/2211.11616/) (2022)
- [The challenge of redundancy on multi-agent value factorisation](https://mlsanity.com/paper/2304.00009/) (2023)
- [Boosting Value Decomposition via Unit-Wise Attentive State Representation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.07182/) (2023)
- [Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertise](https://mlsanity.com/paper/2507.18867/) (2025)
- [Offsetting Unequal Competition through RL-assisted Incentive Schemes](https://mlsanity.com/paper/2201.01450/) (2022)
- [Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.06872/) (2023)
- [SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2503.01458/) (2025)
- [Concept Learning for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2507.20143/) (2025)
- [SafeDiSTL: Decomposing Joint Constraints for Safe Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2610.13737/) (2026)

## Abstract (arXiv)

Recently, deep multiagent reinforcement learning (MARL) has become a highly active research area as many real-world problems can be inherently viewed as multiagent systems. A particularly interesting and widely applicable class of problems is the partially observable cooperative multiagent setting, in which a team of agents learns to coordinate their behaviors conditioning on their private observations and commonly shared global reward signals. One natural solution is to resort to the centralized training and decentralized execution paradigm. During centralized training, one key challenge is the multiagent credit assignment: how to allocate the global rewards for individual agent policies for better coordination towards maximizing system-level's benefits. In this paper, we propose a new method called Q-value Path Decomposition (QPD) to decompose the system's global Q-values into individual agents' Q-values. Unlike previous works which restrict the representation relation of the individual Q-values and the global one, we leverage the integrated gradient attribution technique into deep MARL to directly decompose global Q-values along trajectory paths to assign credits for agents. We evaluate QPD on the challenging StarCraft II micromanagement tasks and show that QPD achieves the state-of-the-art performance in both homogeneous and heterogeneous multiagent scenarios compared with existing cooperative MARL algorithms.