# PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration

arXiv: https://arxiv.org/abs/2203.08553 · HTML: https://mlsanity.com/paper/2203.08553/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 30 of 30)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [RODE: Learning Roles to Decompose Multi-Agent Tasks](https://mlsanity.com/paper/2010.01523/) (2020)
- [Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games](https://mlsanity.com/paper/1703.10069/) (2017)
- [Learning Latent Representations to Influence Multi-Agent Interaction](https://mlsanity.com/paper/2011.06619/) (2020)
- [Influence-Based Multi-Agent Exploration](https://mlsanity.com/paper/1910.05512/) (2019)
- [Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1901.09207/) (2019)
- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1901.11454/) (2019)
- [Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control](https://mlsanity.com/paper/1909.02682/) (2019)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Generative Adversarial Self-Imitation Learning](https://mlsanity.com/paper/1812.00950/) (2018)
- [Action Semantics Network: Considering the Effects of Actions in Multiagent Systems](https://mlsanity.com/paper/1907.11461/) (2019)
- [Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08647/) (2018)
- [Multi-Agent Interactions Modeling with Correlated Policies](https://mlsanity.com/paper/2001.03415/) (2020)
- [KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge](https://mlsanity.com/paper/2002.07418/) (2020)
- [A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.02732/) (2020)
- [A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.01221/) (2022)
- [Continuous Multiagent Control using Collective Behavior Entropy for Large-Scale Home Energy Management](https://mlsanity.com/paper/2005.10000/) (2020)
- [Weighted Double Deep Multiagent Reinforcement Learning in Stochastic Cooperative Environments](https://mlsanity.com/paper/1802.08534/) (2018)
- [Signal Instructed Coordination in Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1909.04224/) (2019)

## Built on by (most cited first, 23 of 23)

- [MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL](https://mlsanity.com/paper/2305.19923/) (2023)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [LLM-Powered Decentralized Generative Agents with Adaptive Hierarchical Knowledge Graph for Cooperative Planning](https://mlsanity.com/paper/2502.05453/) (2025)
- [Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influence](https://mlsanity.com/paper/2302.03322/) (2023)
- [GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis](https://mlsanity.com/paper/2205.13728/) (2022)
- [Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2312.09539/) (2023)
- [EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model](https://mlsanity.com/paper/2210.00498/) (2022)
- [Position: Towards a Responsible LLM-empowered Multi-Agent Systems](https://mlsanity.com/paper/2502.01714/) (2025)
- [Robust Multi-Agent Reinforcement Learning by Mutual Information Regularization](https://mlsanity.com/paper/2310.09833/) (2023)
- [Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing](https://mlsanity.com/paper/2402.02097/) (2024)
- [Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.09032/) (2023)
- [Optimistic Multi-Agent Policy Gradient](https://mlsanity.com/paper/2311.01953/) (2023)
- [Multi-Agent Continuous Control with Generative Flow Networks](https://mlsanity.com/paper/2408.06920/) (2024)
- [Multi-Agent Reinforcement Learning with Communication-Constrained Priors](https://mlsanity.com/paper/2512.03528/) (2025)
- [SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks](https://mlsanity.com/paper/2503.11726/) (2025)
- [Focusing Influence Mechanism for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.19417/) (2025)
- [Mastering Zero-Shot Interactions in Cooperative and Competitive Simultaneous Games](https://mlsanity.com/paper/2402.03136/) (2024)
- [Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.18024/) (2026)
- [GHQ: Grouped Hybrid Q Learning for Heterogeneous Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2303.01070/) (2023)
- [Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2505.08630/) (2025)
- [Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies](https://mlsanity.com/paper/2508.01049/) (2025)
- [MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2605.01805/) (2026)
- [Federated Agent Optimization](https://mlsanity.com/paper/2610.01195/) (2026)

## Abstract (arXiv)

Learning to collaborate is critical in Multi-Agent Reinforcement Learning (MARL). Previous works promote collaboration by maximizing the correlation of agents' behaviors, which is typically characterized by Mutual Information (MI) in different forms. However, we reveal sub-optimal collaborative behaviors also emerge with strong correlations, and simply maximizing the MI can, surprisingly, hinder the learning towards better collaboration. To address this issue, we propose a novel MARL framework, called Progressive Mutual Information Collaboration (PMIC), for more effective MI-driven collaboration. PMIC uses a new collaboration criterion measured by the MI between global states and joint actions. Based on this criterion, the key idea of PMIC is maximizing the MI associated with superior collaborative behaviors and minimizing the MI associated with inferior ones. The two MI objectives play complementary roles by facilitating better collaborations while avoiding falling into sub-optimal ones. Experiments on a wide range of MARL benchmarks show the superior performance of PMIC compared with other algorithms.