# Counterfactual Influence in Markov Decision Processes

arXiv: https://arxiv.org/abs/2402.08514 · HTML: https://mlsanity.com/paper/2402.08514/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review](https://mlsanity.com/paper/2010.10596/) (2020)
- [A* Sampling](https://mlsanity.com/paper/1411.0030/) (2014)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Causal Bandits: Learning Good Interventions via Causal Inference](https://mlsanity.com/paper/1606.03203/) (2016)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [Partial Counterfactual Identification from Observational and Experimental Data](https://mlsanity.com/paper/2110.05690/) (2021)
- [Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation](https://mlsanity.com/paper/2012.09092/) (2020)
- [Causal Reasoning from Meta-reinforcement Learning](https://mlsanity.com/paper/1901.08162/) (2019)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Explainable Planning](https://mlsanity.com/paper/1709.10256/) (2017)
- [Counterfactual Explanations in Sequential Decision Making Under Uncertainty](https://mlsanity.com/paper/2107.02776/) (2021)
- [Counterfactual Temporal Point Processes](https://mlsanity.com/paper/2111.07603/) (2021)
- [Contrastive Explanations of Plans Through Model Restrictions](https://mlsanity.com/paper/2103.15575/) (2021)
- [Learning Generalized Gumbel-max Causal Mechanisms](https://mlsanity.com/paper/2111.06888/) (2021)
- [Counterfactually Guided Off-policy Transfer in Clinical Settings](https://mlsanity.com/paper/2006.11654/) (2020)
- [Counterfactual Inference of Second Opinions](https://mlsanity.com/paper/2203.08653/) (2022)
- [Finding Counterfactually Optimal Action Sequences in Continuous State Spaces](https://mlsanity.com/paper/2306.03929/) (2023)
- [Towards Providing Explanations for AI Planner Decisions](https://mlsanity.com/paper/1810.06338/) (2018)
- [Artificial Intelligence Framework for Simulating Clinical Decision-Making: A Markov Decision Process Approach](https://mlsanity.com/paper/1301.2158/) (2013)
- [ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policies](https://mlsanity.com/paper/2402.06503/) (2024)
- [Causal Temporal Reasoning for Markov Decision Processes](https://mlsanity.com/paper/2212.08712/) (2022)
- [Intervention Efficient Algorithm for Two-Stage Causal MDPs](https://mlsanity.com/paper/2111.00886/) (2021)

## Built on by (most cited first, 5 of 5)

- [Robust Counterfactual Inference in Markov Decision Processes](https://mlsanity.com/paper/2502.13731/) (2025)
- [Counterfactual Strategies for Markov Decision Processes](https://mlsanity.com/paper/2505.09412/) (2025)
- [Counterfactual Scenarios for Automated Planning](https://mlsanity.com/paper/2508.21521/) (2025)
- [Attribution-based Explanations for Markov Decision Processes](https://mlsanity.com/paper/2605.09780/) (2026)
- [Robust Counterfactuals for Markov Decision Processes under Statistical and Causal Uncertainties](https://mlsanity.com/paper/2610.20210/) (2026)

## Abstract (arXiv)

Our work addresses a fundamental problem in the context of counterfactual inference for Markov Decision Processes (MDPs). Given an MDP path τ, this kind of inference allows us to derive counterfactual paths τ' describing what-if versions of τ obtained under different action sequences than those observed in τ. However, as the counterfactual states and actions deviate from the observed ones over time, the observation τ may no longer influence the counterfactual world, meaning that the analysis is no longer tailored to the individual observation, resulting in interventional outcomes rather than counterfactual ones. Even though this issue specifically affects the popular Gumbel-max structural causal model used for MDP counterfactuals, it has remained overlooked until now. In this work, we introduce a formal characterisation of influence based on comparing counterfactual and interventional distributions. We devise an algorithm to construct counterfactual models that automatically satisfy influence constraints. Leveraging such models, we derive counterfactual policies that are not just optimal for a given reward structure but also remain tailored to the observed path. Even though there is an unavoidable trade-off between policy optimality and strength of influence constraints, our experiments demonstrate that it is possible to derive (near-)optimal policies while remaining under the influence of the observation.