# A Roadmap Towards Improving Multi-Agent Reinforcement Learning With Causal Discovery And Inference

arXiv: https://arxiv.org/abs/2503.17803 · HTML: https://mlsanity.com/paper/2503.17803/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 22 Mar 2025.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 29 of 29)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Safe and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1606.02647/) (2016)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Causal-learn: Causal Discovery in Python](https://mlsanity.com/paper/2307.16405/) (2023)
- [DiBS: Differentiable Bayesian Structure Learning](https://mlsanity.com/paper/2105.11839/) (2021)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning](https://mlsanity.com/paper/2207.03530/) (2022)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Nonlinear Invariant Risk Minimization: A Causal Approach](https://mlsanity.com/paper/2102.12353/) (2021)
- [Heterogeneous Multi-Robot Reinforcement Learning](https://mlsanity.com/paper/2301.07137/) (2023)
- [BenchMARL: Benchmarking Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2312.01472/) (2023)
- [Sample-Efficient Reinforcement Learning of Partially Observable Markov Games](https://mlsanity.com/paper/2206.01315/) (2022)
- [Multi-Agent Causal Discovery Using Large Language Models](https://mlsanity.com/paper/2407.15073/) (2024)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [Causal Multi-Agent Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2111.06721/) (2021)
- [Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning](https://mlsanity.com/paper/2406.14302/) (2024)
- [Provably Efficient Causal Model-Based Reinforcement Learning for Systematic Generalization](https://mlsanity.com/paper/2202.06545/) (2022)
- [Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2312.09539/) (2023)
- [Algorithms for Causal Reasoning in Probability Trees](https://mlsanity.com/paper/2010.12237/) (2020)
- [MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment](https://mlsanity.com/paper/2312.03644/) (2023)
- [Exploiting Causal Graph Priors with Posterior Sampling for Reinforcement Learning](https://mlsanity.com/paper/2310.07518/) (2023)
- [Challenges and Considerations in the Evaluation of Bayesian Causal Discovery](https://mlsanity.com/paper/2406.03209/) (2024)
- [Incremental effects for continuous exposures](https://mlsanity.com/paper/2409.11967/) (2024)
- [Learning Representations of Instruments for Partial Identification of Treatment Effects](https://mlsanity.com/paper/2410.08976/) (2024)
- [Discovering Minimal Reinforcement Learning Environments](https://mlsanity.com/paper/2406.12589/) (2024)
- [Discovering Causality for Efficient Cooperation in Multi-Agent Environments](https://mlsanity.com/paper/2306.11846/) (2023)

## Built on by (most cited first, 1 of 1)

- [Causal Reflection with Language Models](https://mlsanity.com/paper/2508.04495/) (2025)

## Abstract (arXiv)

Causal reasoning is increasingly used in Reinforcement Learning (RL) to improve the learning process in several dimensions: efficacy of learned policies, efficiency of convergence, generalisation capabilities, safety and interpretability of behaviour. However, applications of causal reasoning to Multi-Agent RL (MARL) are still mostly unexplored. In this paper, we take the first step in investigating the opportunities and challenges of applying causal reasoning in MARL. We measure the impact of a simple form of causal augmentation in state-of-the-art MARL scenarios increasingly requiring cooperation, and with state-of-the-art MARL algorithms exploiting various degrees of collaboration between agents. Then, we discuss the positive as well as negative results achieved, giving us the chance to outline the areas where further research may help to successfully transfer causal RL to the multi-agent setting.