# Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming

arXiv: https://arxiv.org/abs/2201.08484 · HTML: https://mlsanity.com/paper/2201.08484/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 70 references that are in the map.
- First posted on arXiv 20 Jan 2022.

## Builds on (most cited first, 25 of 25)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [TarMAC: Targeted Multi-Agent Communication](https://mlsanity.com/paper/1810.11187/) (2018)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Learning to Schedule Communication in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1902.01554/) (2019)
- [Multi-Agent Game Abstraction via Graph Attention Neural Network](https://mlsanity.com/paper/1911.10715/) (2019)
- [Influence-Based Multi-Agent Exploration](https://mlsanity.com/paper/1910.05512/) (2019)
- [Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1901.09207/) (2019)
- [Byzantine-Resilient Non-Convex Stochastic Gradient Descent](https://mlsanity.com/paper/2012.14368/) (2020)
- [Lenient Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1707.04402/) (2017)
- [CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1809.05188/) (2018)
- [Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning](https://mlsanity.com/paper/1901.09216/) (2019)
- [VIREL: A Variational Inference Framework for Reinforcement Learning](https://mlsanity.com/paper/1811.01132/) (2018)
- [A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.02732/) (2020)
- [Byzantine-Robust Decentralized Stochastic Optimization over Static and Time-Varying Networks](https://mlsanity.com/paper/2005.06276/) (2020)
- [Coordinated Control of UAVs for Human-Centered Active Sensing of Wildfires](https://mlsanity.com/paper/2006.07969/) (2020)
- [Heterogeneous Graph Attention Networks for Learning Diverse Communication](https://mlsanity.com/paper/2108.09568/) (2021)
- [Safe Coordination of Human-Robot Firefighting Teams](https://mlsanity.com/paper/1903.06847/) (2019)
- [Bounded-Rational Pursuit-Evasion Games](https://mlsanity.com/paper/2003.06954/) (2020)

## Built on by (most cited first, 14 of 14)

- [Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2209.11908/) (2022)
- [Multi-Agent Coordination via Multi-Level Communication](https://mlsanity.com/paper/2209.12713/) (2022)
- [Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference](https://mlsanity.com/paper/2412.14355/) (2024)
- [DM2: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching](https://mlsanity.com/paper/2206.00233/) (2022)
- [Mediated Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.08419/) (2023)
- [Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey](https://mlsanity.com/paper/2401.04934/) (2024)
- [Multi-UAV Planning for Cooperative Wildfire Coverage and Tracking with Quality-of-Service Guarantees](https://mlsanity.com/paper/2206.10544/) (2022)
- [Best Possible Q-Learning](https://mlsanity.com/paper/2302.01188/) (2023)
- [Learning to Negotiate via Voluntary Commitment](https://mlsanity.com/paper/2503.03866/) (2025)
- [Strategy Discovery and Mixture in Lifelong Learning from Heterogeneous Demonstration](https://mlsanity.com/paper/2202.07014/) (2022)
- [TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient](https://mlsanity.com/paper/2312.15667/) (2023)
- [Off-Policy Action Anticipation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.01447/) (2023)
- [Towards In-Vehicle Multi-Task Facial Attribute Recognition: Investigating Synthetic Data and Vision Foundation Models](https://mlsanity.com/paper/2403.06088/) (2024)
- [Realtime Dynamic Gaze Target Tracking and Depth-Level Estimation](https://mlsanity.com/paper/2406.18595/) (2024)

## Abstract (arXiv)

Information sharing is key in building team cognition and enables coordination and cooperation. High-performing human teams also benefit from acting strategically with hierarchical levels of iterated communication and rationalizability, meaning a human agent can reason about the actions of their teammates in their decision-making. Yet, the majority of prior work in Multi-Agent Reinforcement Learning (MARL) does not support iterated rationalizability and only encourage inter-agent communication, resulting in a suboptimal equilibrium cooperation strategy. In this work, we show that reformulating an agent's policy to be conditional on the policies of its neighboring teammates inherently maximizes Mutual Information (MI) lower-bound when optimizing under Policy Gradient (PG). Building on the idea of decision-making under bounded rationality and cognitive hierarchy theory, we show that our modified PG approach not only maximizes local agent rewards but also implicitly reasons about MI between agents without the need for any explicit ad-hoc regularization terms. Our approach, InfoPG, outperforms baselines in learning emergent collaborative behaviors and sets the state-of-the-art in decentralized cooperative MARL tasks. Our experiments validate the utility of InfoPG by achieving higher sample efficiency and significantly larger cumulative reward in several complex cooperative multi-agent domains.