# Causal Temporal Reasoning for Markov Decision Processes

arXiv: https://arxiv.org/abs/2212.08712 · HTML: https://mlsanity.com/paper/2212.08712/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 16 Dec 2022.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Counterfactual Explanations in Sequential Decision Making Under Uncertainty](https://mlsanity.com/paper/2107.02776/) (2021)
- [Deconfounding Reinforcement Learning in Observational Settings](https://mlsanity.com/paper/1812.10576/) (2018)
- [From Verification to Causality-based Explications](https://mlsanity.com/paper/2105.09533/) (2021)
- [Probabilistic Hyperproperties of Markov Decision Processes](https://mlsanity.com/paper/2005.03362/) (2020)
- [Probabilistic causes in Markov chains](https://mlsanity.com/paper/2104.13604/) (2021)
- [On probability-raising causality in Markov decision processes](https://mlsanity.com/paper/2201.08768/) (2022)
- [Probabilistic Hyperproperties with Nondeterminism](https://mlsanity.com/paper/2005.06115/) (2020)

## Built on by (most cited first, 3 of 3)

- [Counterfactual Influence in Markov Decision Processes](https://mlsanity.com/paper/2402.08514/) (2024)
- [Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees](https://mlsanity.com/paper/2606.29681/) (2026)
- [Property-driven Causal Abstractions for Markov Decision Processes](https://mlsanity.com/paper/2607.26787/) (2026)

## Abstract (arXiv)

We introduce textitPCFTL (Probabilistic CounterFactual Temporal Logic), a new probabilistic temporal logic for the verification of Markov Decision Processes (MDP). PCFTL is the first to include operators for causal reasoning, allowing us to express interventional and counterfactual queries. Given a path formula φ, an interventional property is concerned with the satisfaction probability of φ if we apply a particular change I to the MDP (e.g., switching to a different policy); a counterfactual allows us to compute, given an observed MDP path τ, what the outcome of φ would have been had we applied I in the past. For its ability to reason about textitwhat-if scenarios involving different configurations of the MDP, our approach represents a departure from existing probabilistic temporal logics that can only reason about a fixed system configuration. From a syntactic viewpoint, we introduce a generalized counterfactual operator that subsumes both interventional and counterfactual probabilities as well as the traditional probabilistic operator found in e.g., PCTL. From a semantics viewpoint, our logic is interpreted over a structural causal model translation of the MDP, which gives us a representation amenable to counterfactual reasoning. We evaluate PCFTL in the context of safe reinforcement learning using a benchmark of grid-world models.