# Causal Multi-Agent Reinforcement Learning: Review and Open Problems

arXiv: https://arxiv.org/abs/2111.06721 · HTML: https://mlsanity.com/paper/2111.06721/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 98 references that are in the map.
- First posted on arXiv 12 Nov 2021.

## Builds on (most cited first, 30 of 38)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [TarMAC: Targeted Multi-Agent Communication](https://mlsanity.com/paper/1810.11187/) (2018)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [Causal Discovery with Reinforcement Learning](https://mlsanity.com/paper/1906.04477/) (2019)
- [ROMA: Multi-Agent Reinforcement Learning with Emergent Roles](https://mlsanity.com/paper/2003.08039/) (2020)
- [Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data](https://mlsanity.com/paper/2006.10833/) (2020)
- [Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1906.04737/) (2019)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.03400/) (2021)
- [External Validity: From Do-Calculus to Transportability Across Populations](https://mlsanity.com/paper/1503.01603/) (2015)
- [Modeling Others using Oneself in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.09640/) (2018)
- [Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.09756/) (2018)
- [Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding](https://mlsanity.com/paper/2003.05623/) (2020)
- [Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward](https://mlsanity.com/paper/2006.06626/) (2020)
- [Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding](https://mlsanity.com/paper/2103.04850/) (2021)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [Learning to Teach in Cooperative Multiagent Reinforcement Learning](https://mlsanity.com/paper/1805.07830/) (2018)
- [From Ordinary Differential Equations to Structural Causal Models: the deterministic case](https://mlsanity.com/paper/1408.2063/) (2014)
- [Causal Reinforcement Learning using Observational and Interventional Data](https://mlsanity.com/paper/2106.14421/) (2021)
- [Rethinking Formal Models of Partially Observable Multiagent Decision Making](https://mlsanity.com/paper/1906.11110/) (2019)
- [Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling](https://mlsanity.com/paper/2006.07178/) (2020)
- [On Inductive Biases in Deep Reinforcement Learning](https://mlsanity.com/paper/1907.02908/) (2019)
- [Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training](https://mlsanity.com/paper/2107.14316/) (2021)

## Built on by (most cited first, 16 of 16)

- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Causal Composition Diffusion Model for Closed-loop Traffic Generation](https://mlsanity.com/paper/2412.17920/) (2024)
- [Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach](https://mlsanity.com/paper/2305.18427/) (2023)
- [MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment](https://mlsanity.com/paper/2312.03644/) (2023)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Causality from Bottom to Top: A Survey](https://mlsanity.com/paper/2403.11219/) (2024)
- [Discovering Causality for Efficient Cooperation in Multi-Agent Environments](https://mlsanity.com/paper/2306.11846/) (2023)
- [Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlations](https://mlsanity.com/paper/2507.09742/) (2025)
- [Unifying Agent Interaction and World Information for Multi-agent Coordination](https://mlsanity.com/paper/2509.25550/) (2025)
- [Using Contrastive Samples for Identifying and Leveraging Possible Causal Relationships in Reinforcement Learning](https://mlsanity.com/paper/2210.17296/) (2022)
- [A Roadmap Towards Improving Multi-Agent Reinforcement Learning With Causal Discovery And Inference](https://mlsanity.com/paper/2503.17803/) (2025)
- [Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications](https://mlsanity.com/paper/2512.18135/) (2025)
- [Wind Estimation in Unmanned Aerial Vehicles with Causal Machine Learning](https://mlsanity.com/paper/2407.01154/) (2024)
- [Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges](https://mlsanity.com/paper/2505.08453/) (2025)
- [CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learning](https://mlsanity.com/paper/2602.09207/) (2026)
- [CRRL: A Causality-Based Reinforcement Learning Framework for Autonomous System Recovery](https://mlsanity.com/paper/2607.03177/) (2026)

## Abstract (arXiv)

This paper serves to introduce the reader to the field of multi-agent reinforcement learning (MARL) and its intersection with methods from the study of causality. We highlight key challenges in MARL and discuss these in the context of how causal methods may assist in tackling them. We promote moving toward a 'causality first' perspective on MARL. Specifically, we argue that causality can offer improved safety, interpretability, and robustness, while also providing strong theoretical guarantees for emergent behaviour. We discuss potential solutions for common challenges, and use this context to motivate future research directions.