mlsanityData to 7 Oct 2026

Causal Multi-Agent Reinforcement Learning: Review and Open Problems

arXiv 2111.06721 ↗First posted 12 Nov 2021cs.LG cs.AI stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper serves to introduce the reader to the field of multi-agent reinforcement learning (MARL) and its intersection with methods from the study of causality. We highlight key challenges in MARL and discuss these in the context of how causal methods may assist in tackling them. We promote moving toward a 'causality first' perspective on MARL. Specifically, we argue that causality can offer improved safety, interpretability, and robustness, while also providing strong theoretical guarantees for emergent behaviour. We discuss potential solutions for common challenges, and use this context to motivate future research directions.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 38 of its 98 references that are in the map. First posted on arXiv 12 Nov 2021.

Builds on

38 of 98 references.

References · 38

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2018World Modelscited by 1,444
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  4. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  5. 2020The Future of Digital Health with Federated Learningcited by 625
  6. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  7. 2019Causality for Machine Learningcited by 242
  8. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  9. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  10. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  11. 2019Causal Discovery with Reinforcement Learningcited by 145
  12. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 135
26 more
  1. 2020Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Datacited by 99
  2. 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 90
  3. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 81
  4. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  5. 2015External Validity: From Do-Calculus to Transportability Across Populationscited by 75
  6. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  7. 2018Modeling Others using Oneself in Multi-Agent Reinforcement Learningcited by 64
  8. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  9. 2021Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confoundingcited by 50
  10. 2020Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Rewardcited by 46
  11. 2018Learning to Teach in Cooperative Multiagent Reinforcement Learningcited by 45
  12. 2019Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Surveycited by 45
  13. 2014From Ordinary Differential Equations to Structural Causal Models: the deterministic casecited by 41
  14. 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 38
  15. 2019Rethinking Formal Models of Partially Observable Multiagent Decision Makingcited by 35
  16. 2020Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabelingcited by 24
  17. 2019On Inductive Biases in Deep Reinforcement Learningcited by 22
  18. 2021Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Trainingcited by 16
  19. 2019Teaching on a Budget in Multi-Agent Deep Reinforcement Learningcited by 14
  20. 2020Learning to Communicate Using Counterfactual Reasoningcited by 9
  21. 2021Asymptotic Theory for IV-Based Reinforcement Learning with Potential Endogeneitycited by 8
  22. 2021Causal Markov Decision Processes: Learning Good Interventions Efficientlycited by 7
  23. 2020A Critical View of the Structural Causal Modelcited by 6
  24. 2021Robust and Scalable Routing with Multi-Agent Deep Reinforcement Learning for MANETscited by 5
  25. 2019Causal Games and Causal Nash Equilibriumcited by 2
  26. 2019Reinforcement Learning is not a Causal problemcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 3; 2025: 5; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023A Survey on Causal Reinforcement Learningcited by 43
  2. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  3. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  4. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  5. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  6. 2024Causality from Bottom to Top: A Surveycited by 5
  7. 2023Discovering Causality for Efficient Cooperation in Multi-Agent Environmentscited by 2
  8. 2025Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlationscited by 2
  9. 2022Using Contrastive Samples for Identifying and Leveraging Possible Causal Relationships in Reinforcement Learningcited by 1
  10. 2025A Roadmap Towards Improving Multi-Agent Reinforcement Learning With Causal Discovery And Inferencecited by 1
  11. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 1
  12. 2025Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applicationscited by 1
4 more
  1. 2024Wind Estimation in Unmanned Aerial Vehicles with Causal Machine Learningcited by 0
  2. 2025Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challengescited by 0
  3. 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learningcited by 0
  4. 2026CRRL: A Causality-Based Reinforcement Learning Framework for Autonomous System Recoverycited by 0