# Robust Counterfactual Inference in Markov Decision Processes

arXiv: https://arxiv.org/abs/2502.13731 · HTML: https://mlsanity.com/paper/2502.13731/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [Counterfactual Probabilities: Computational Methods, Bounds and Applications](https://mlsanity.com/paper/1302.6784/) (2013)
- [Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation](https://mlsanity.com/paper/2012.09092/) (2020)
- [An Automated Approach to Causal Inference in Discrete Settings](https://mlsanity.com/paper/2109.13471/) (2021)
- [Structural Causal Models Are (Solvable by) Credal Networks](https://mlsanity.com/paper/2008.00463/) (2020)
- [Estimating Categorical Counterfactuals via Deep Twin Networks](https://mlsanity.com/paper/2109.01904/) (2021)
- [Counterfactual (Non-)identifiability of Learned Structural Causal Models](https://mlsanity.com/paper/2301.09031/) (2023)
- [Learning Generalized Gumbel-max Causal Mechanisms](https://mlsanity.com/paper/2111.06888/) (2021)
- [Counterfactually Guided Off-policy Transfer in Clinical Settings](https://mlsanity.com/paper/2006.11654/) (2020)
- [Finding Counterfactually Optimal Action Sequences in Continuous State Spaces](https://mlsanity.com/paper/2306.03929/) (2023)
- [Causal Expectation-Maximisation](https://mlsanity.com/paper/2011.02912/) (2020)
- [Approximating Counterfactual Bounds while Fusing Observational, Biased and Randomised Data Sources](https://mlsanity.com/paper/2307.16577/) (2023)
- [Counterfactual Influence in Markov Decision Processes](https://mlsanity.com/paper/2402.08514/) (2024)
- [ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policies](https://mlsanity.com/paper/2402.06503/) (2024)

## Built on by (most cited first, 3 of 3)

- [Property-driven Causal Abstractions for Markov Decision Processes](https://mlsanity.com/paper/2607.26787/) (2026)
- [Learning to Control Coupled-Dynamics Environments with Joint Markov Decision Processes](https://mlsanity.com/paper/2608.22765/) (2026)
- [Same Predictions, Different Harms: Causal Auditing of Patient World Models](https://mlsanity.com/paper/2610.05198/) (2026)

## Abstract (arXiv)

This paper addresses a key limitation in existing counterfactual inference methods for Markov Decision Processes (MDPs). Current approaches assume a specific causal model to make counterfactuals identifiable. However, there are usually many causal models that align with the observational and interventional distributions of an MDP, each yielding different counterfactual distributions, so fixing a particular causal model limits the validity (and usefulness) of counterfactual inference. We propose a novel non-parametric approach that computes tight bounds on counterfactual transition probabilities across all compatible causal models. Unlike previous methods that require solving prohibitively large optimisation problems (with variables that grow exponentially in the size of the MDP), our approach provides closed-form expressions for these bounds, making computation highly efficient and scalable for non-trivial MDPs. Once such an interval counterfactual MDP is constructed, our method identifies robust counterfactual policies that optimise the worst-case reward w.r.t. the uncertain interval MDP probabilities. We evaluate our method on various case studies, demonstrating improved robustness over existing methods.