Discovering Causality for Efficient Cooperation in Multi-Agent Environments
Not on the map. Why?
Abstract (arXiv)
In cooperative Multi-Agent Reinforcement Learning (MARL) agents are required to learn behaviours as a team to achieve a common goal. However, while learning a task, some agents may end up learning sub-optimal policies, not contributing to the objective of the team. Such agents are called lazy agents due to their non-cooperative behaviours that may arise from failing to understand whether they caused the rewards. As a consequence, we observe that the emergence of cooperative behaviours is not necessarily a byproduct of being able to solve a task as a team. In this paper, we investigate the applications of causality in MARL and how it can be applied in MARL to penalise these lazy agents. We observe that causality estimations can be used to improve the credit assignment to the agents and show how it can be leveraged to improve independent learning in MARL. Furthermore, we investigate how Amortized Causal Discovery can be used to automate causality detection within MARL environments. The results demonstrate that causality relations between individual observations and the team reward can be used to detect and punish lazy agents, making them develop more intelligent behaviours. This results in improvements not only in the overall performances of the team but also in their individual capabilities. In addition, results show that Amortized Causal Discovery can be used efficiently to find causal relations in MARL.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 20 of its 43 references that are in the map. First posted on arXiv 20 Jun 2023.
Builds on
20 of 43 references.
References · 20
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2017Counterfactual Multi-Agent Policy Gradientscited by 751
- 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
- 2019The StarCraft Multi-Agent Challengecited by 597
- 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
- 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
- 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
- 2016Learning Multiagent Communication with Backpropagationcited by 366
- 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
- 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 184
- 2019Causal Discovery with Reinforcement Learningcited by 145
- 2020Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Datacited by 100
8 more
- 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 93
- 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 87
- 2019Shapley Q-value: A Local Reward Approach to Solve Global Reward Gamescited by 73
- 2019Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causalitycited by 55
- 2020Reward Machines for Cooperative Multi-Agent Reinforcement Learningcited by 35
- 2018Multi-Agent Common Knowledge Reinforcement Learningcited by 31
- 2021Causal Multi-Agent Reinforcement Learning: Review and Open Problemscited by 16
- 2020Causal Discovery from Incomplete Data using An Encoder and Reinforcement Learningcited by 7
Built on by
Citing papers per year: 2025: 2.
2 papers.