# Reasoning about Causality in Games

arXiv: https://arxiv.org/abs/2301.02324 · HTML: https://mlsanity.com/paper/2301.02324/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 7 of its 39 references that are in the map.
- First posted on arXiv 5 Jan 2023.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 7 of 7)

- [Fair Inference On Outcomes](https://mlsanity.com/paper/1705.10378/) (2017)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [Blameworthiness in Multi-Agent Settings](https://mlsanity.com/paper/1903.04102/) (2019)
- [How RL Agents Behave When Their Actions Are Modified](https://mlsanity.com/paper/2102.07716/) (2021)
- [Why Fair Labels Can Yield Unfair Predictions: Graphical Conditions for Introduced Unfairness](https://mlsanity.com/paper/2202.10816/) (2022)
- [Networks of Influence Diagrams: A Formalism for Representing Agents' Beliefs and Decision-Making Processes](https://mlsanity.com/paper/1401.3426/) (2014)

## Built on by (most cited first, 20 of 20)

- [Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents](https://mlsanity.com/paper/2505.02077/) (2025)
- [Honesty Is the Best Policy: Defining and Mitigating AI Deception](https://mlsanity.com/paper/2312.01350/) (2023)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [Causal Reasoning: Charting a Revolutionary Course for Next-Generation AI-Native Wireless Networks](https://mlsanity.com/paper/2309.13223/) (2023)
- [From Imitation to Introspection: Probing Self-Consciousness in Language Models](https://mlsanity.com/paper/2410.18819/) (2024)
- [ExoPredicator: Learning Abstract Models of Dynamic Worlds for Robot Planning](https://mlsanity.com/paper/2509.26255/) (2025)
- [MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment](https://mlsanity.com/paper/2312.03644/) (2023)
- [Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety](https://mlsanity.com/paper/2305.19223/) (2023)
- [On Imperfect Recall in Multi-Agent Influence Diagrams](https://mlsanity.com/paper/2307.05059/) (2023)
- [Characterising Decision Theories with Mechanised Causal Graphs](https://mlsanity.com/paper/2307.10987/) (2023)
- [A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.17697/) (2025)
- [From Gameplay Traces to Game Mechanics: Causal Induction with Large Language Models](https://mlsanity.com/paper/2602.00190/) (2026)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)
- [Experiments with Detecting and Mitigating AI Deception](https://mlsanity.com/paper/2306.14816/) (2023)
- [Characterising Interventions in Causal Games](https://mlsanity.com/paper/2406.09318/) (2024)
- [Graphical Models for Decision-Making: Integrating Causality and Game Theory](https://mlsanity.com/paper/2504.13210/) (2025)
- [Causal Foundations of Collective Agency](https://mlsanity.com/paper/2605.00248/) (2026)
- [A Knowledge-Based Multi-Agent Framework for Security Control Recommendation](https://mlsanity.com/paper/2607.09954/) (2026)
- [Equilibrium Causal Games: Separation, Identification, and the Identifiability of Cyclic Latent States](https://mlsanity.com/paper/2607.19531/) (2026)
- [Compiling VGDL into Causal Models](https://mlsanity.com/paper/2609.05459/) (2026)

## Abstract (arXiv)

Causal reasoning and game-theoretic reasoning are fundamental topics in artificial intelligence, among many other disciplines: this paper is concerned with their intersection. Despite their importance, a formal framework that supports both these forms of reasoning has, until now, been lacking. We offer a solution in the form of (structural) causal games, which can be seen as extending Pearl's causal hierarchy to the game-theoretic domain, or as extending Koller and Milch's multi-agent influence diagrams to the causal domain. We then consider three key questions: i) How can the (causal) dependencies in games - either between variables, or between strategies - be modelled in a uniform, principled manner? ii) How may causal queries be computed in causal games, and what assumptions does this require? iii) How do causal games compare to existing formalisms? To address question i), we introduce mechanised games, which encode dependencies between agents' decision rules and the distributions governing the game. In response to question ii), we present definitions of predictions, interventions, and counterfactuals, and discuss the assumptions required for each. Regarding question iii), we describe correspondences between causal games and other formalisms, and explain how causal games can be used to answer queries that other causal or game-theoretic models do not support. Finally, we highlight possible applications of causal games, aided by an extensive open-source Python library.