mlsanityData to 14 Oct 2026

Finding Counterfactually Optimal Action Sequences in Continuous State Spaces

arXiv 2306.03929 ↗First posted 6 Jun 2023cs.LG cs.AI cs.CYcited by 13

Not on the map. Why?

Abstract (arXiv)

Whenever a clinician reflects on the efficacy of a sequence of treatment decisions for a patient, they may try to identify critical time steps where, had they made different decisions, the patient's health would have improved. While recent methods at the intersection of causal inference and reinforcement learning promise to aid human experts, as the clinician above, to retrospectively analyze sequential decision making processes, they have focused on environments with finitely many discrete states. However, in many practical applications, the state of the environment is inherently continuous in nature. In this paper, we aim to fill this gap. We start by formally characterizing a sequence of discrete actions and continuous states using finite horizon Markov decision processes and a broad class of bijective structural causal models. Building upon this characterization, we formalize the problem of finding counterfactually optimal action sequences and show that, in general, we cannot expect to solve it in polynomial time. Then, we develop a search method based on the A^* algorithm that, under a natural form of Lipschitz continuity of the environment's dynamics, is guaranteed to return the optimal solution to the problem. Experiments on real clinical data show that our method is very efficient in practice, and it has the potential to offer interesting insights for sequential decision making tasks.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 35 of its 73 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

35 of 73 references.

References · 35

  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  2. 2019Model-Based Reinforcement Learning for Ataricited by 502
  3. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  4. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  5. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  6. 2018Sorting out Lipschitz function approximationcited by 213
  7. 2020Deep Structural Causal Models for Tractable Counterfactual Inferencecited by 211
  8. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  9. 2020Algorithmic recourse under imperfect causal knowledge: a probabilistic approachcited by 140
  10. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  11. 2020Causal Autoregressive Flowscited by 99
  12. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
23 more
  1. 2022Diffusion Causal Models for Counterfactual Estimationcited by 87
  2. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 82
  3. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  4. 2021BCD Nets: Scalable Variational Approaches for Bayesian Causal Discoverycited by 68
  5. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  6. 2021Partial Counterfactual Identification from Observational and Experimental Datacited by 64
  7. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  8. 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 59
  9. 2022On the Identifiability and Estimation of Causal Location-Scale Noise Modelscited by 56
  10. 2020Decisions, Counterfactual Explanations and Strategic Behaviorcited by 44
  11. 2023Counterfactual Identifiability of Bijective Causal Modelscited by 43
  12. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  13. 2021Nested Counterfactual Identification from Arbitrary Surrogate Experimentscited by 38
  14. 2021Counterfactual Explanations in Sequential Decision Making Under Uncertaintycited by 37
  15. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  16. 2022Counterfactual harmcited by 29
  17. 2021Counterfactual Temporal Point Processescited by 20
  18. 2021Near-Optimal Multi-Perturbation Experimental Design for Causal Structure Learningcited by 17
  19. 2021Dynamic Causal Bayesian Optimizationcited by 17
  20. 2023Counterfactual (Non-)identifiability of Learned Structural Causal Modelscited by 17
  21. 2021Multi-task Learning of Order-Consistent Causal Graphscited by 10
  22. 2020Zooming for Efficient Model-Free Reinforcement Learning in Metric Spacescited by 7
  23. 2021Coarse-Grained Smoothness for RL in Metric Spacescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 4; 2026: 3.

13 papers.

Sort:

By area:

  1. 2025Artificial Intelligence Should Genuinely Support Clinical Reasoning and Decision Making To Bridge the Translational Gapcited by 14
  2. 2025Counterfactual Identifiability via Dynamic Optimal Transportcited by 6
  3. 2024Counterfactual Influence in Markov Decision Processescited by 5
  4. 2024ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policiescited by 4
  5. 2024Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Makingcited by 4
  6. 2025Robust Counterfactual Inference in Markov Decision Processescited by 3
  7. 2024Perfect Counterfactuals in Imperfect Worlds: Modelling Noisy Implementation of Actions in Sequential Algorithmic Recoursecited by 2
  8. 2023Agent-Specific Effects: A Causal Effect Propagation Analysis in Multi-Agent MDPscited by 1
  9. 2024Semifactual Explanations for Reinforcement Learningcited by 1
  10. 2026Conservative Continuous-Time Treatment Optimizationcited by 1
  11. 2025Learning Counterfactual Outcomes Under Rank Preservationcited by 0
  12. 2026Robust Counterfactual Policy Optimisation via Nondeterministic Causal Modelscited by 0
1 more
  1. 2026Robust Counterfactuals for Markov Decision Processes under Statistical and Causal Uncertaintiescited by 0