# Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2007.02529 · HTML: https://mlsanity.com/paper/2007.02529/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers](https://mlsanity.com/paper/2101.08001/) (2021)
- [Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00285/) (2021)
- [SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning](https://mlsanity.com/paper/2105.15013/) (2021)
- [URB -- Urban Routing Benchmark for RL-equipped Connected Autonomous Vehicles](https://mlsanity.com/paper/2505.17734/) (2025)
- [VDFD: Multi-Agent Value Decomposition Framework with Disentangled World Model](https://mlsanity.com/paper/2309.04615/) (2023)
- [Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.12957/) (2021)
- [Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.09034/) (2021)
- [Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures](https://mlsanity.com/paper/2109.09038/) (2021)
- [Multi-Agent Model-Based Reinforcement Learning with Joint State-Action Learned Embeddings](https://mlsanity.com/paper/2602.12520/) (2026)

## Abstract (arXiv)

We present a multi-agent actor-critic method that aims to implicitly address the credit assignment problem under fully cooperative settings. Our key motivation is that credit assignment among agents may not require an explicit formulation as long as (1) the policy gradients derived from a centralized critic carry sufficient information for the decentralized agents to maximize their joint action value through optimal cooperation and (2) a sustained level of exploration is enforced throughout training. Under the centralized training with decentralized execution (CTDE) paradigm, we achieve the former by formulating the centralized critic as a hypernetwork such that a latent state representation is integrated into the policy gradients through its multiplicative association with the stochastic policies; to achieve the latter, we derive a simple technique called adaptive entropy regularization where magnitudes of the entropy gradients are dynamically rescaled based on the current policy stochasticity to encourage consistent levels of exploration. Our algorithm, referred to as LICA, is evaluated on several benchmarks including the multi-agent particle environments and a set of challenging StarCraft II micromanagement tasks, and we show that LICA significantly outperforms previous methods.