# Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Making

arXiv: https://arxiv.org/abs/2410.12539 · HTML: https://mlsanity.com/paper/2410.12539/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 22 of 22)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Towards Efficient Data Valuation Based on the Shapley Value](https://mlsanity.com/paper/1902.10275/) (2019)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Causal Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions of Complex Models](https://mlsanity.com/paper/2011.01625/) (2020)
- [Partial Counterfactual Identification from Observational and Experimental Data](https://mlsanity.com/paper/2110.05690/) (2021)
- [Collaborating with language models for embodied reasoning](https://mlsanity.com/paper/2302.00763/) (2023)
- [Shapley Flow: A Graph-based Approach to Interpreting Model Predictions](https://mlsanity.com/paper/2010.14592/) (2020)
- [Nested Counterfactual Identification from Arbitrary Surrogate Experiments](https://mlsanity.com/paper/2107.03190/) (2021)
- [Counterfactual Explanations in Sequential Decision Making Under Uncertainty](https://mlsanity.com/paper/2107.02776/) (2021)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [Counterfactual harm](https://mlsanity.com/paper/2204.12993/) (2022)
- [Achieving Counterfactual Fairness for Causal Bandit](https://mlsanity.com/paper/2109.10458/) (2021)
- [A Causal Analysis of Harm](https://mlsanity.com/paper/2210.05327/) (2022)
- [Finding Counterfactually Optimal Action Sequences in Continuous State Spaces](https://mlsanity.com/paper/2306.03929/) (2023)
- [Quantifying intrinsic causal contributions via structure preserving interventions](https://mlsanity.com/paper/2007.00714/) (2020)
- [Blameworthiness in Multi-Agent Settings](https://mlsanity.com/paper/1903.04102/) (2019)
- [Actual Causality and Responsibility Attribution in Decentralized Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2204.00302/) (2022)

## Built on by (most cited first, 4 of 4)

- [Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.14892/) (2026)
- [COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams](https://mlsanity.com/paper/2604.17693/) (2026)
- [Adaptive Accountability in Networked MAS: Tracing and Mitigating Emergent Norms at Scale](https://mlsanity.com/paper/2512.18561/) (2025)
- [Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs](https://mlsanity.com/paper/2609.19636/) (2026)

## Abstract (arXiv)

We address the challenge of explaining counterfactual outcomes in multi-agent Markov decision processes. In particular, we aim to explain the total counterfactual effect of an agent's action on the outcome of a realized scenario through its influence on the environment dynamics and the agents' behavior. To achieve this, we introduce a novel causal explanation formula that decomposes the counterfactual effect by attributing to each agent and state variable a score reflecting their respective contributions to the effect. First, we show that the total counterfactual effect of an agent's action can be decomposed into two components: one measuring the effect that propagates through all subsequent agents' actions and another related to the effect that propagates through the state transitions. Building on recent advancements in causal contribution analysis, we further decompose these two effects as follows. For the former, we consider agent-specific effects -- a causal concept that quantifies the counterfactual effect of an agent's action that propagates through a subset of agents. Based on this notion, we use Shapley value to attribute the effect to individual agents. For the latter, we consider the concept of structure-preserving interventions and attribute the effect to state variables based on their "intrinsic" contributions. Through extensive experimentation, we demonstrate the interpretability of our approach in a Gridworld environment with LLM-assisted agents and a sepsis management simulator.