mlsanityData to 14 Oct 2026

Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

arXiv 1811.06272 ↗First posted 15 Nov 2018cs.LG stat.MLcited by 82

Not on the map. Why?

Abstract (arXiv)

Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 13; 2023: 15; 2024: 13; 2025: 14; 2026: 10.

82 papers.

Sort:

By area:

  1. 2021Towards Causal Representation Learningcited by 209
  2. 2021Pathdreamer: A World Model for Indoor Navigationcited by 118
  3. 2021Independent mechanism analysis, a new concept?cited by 97
  4. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
  5. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 65
  6. 2023A Survey on Causal Reinforcement Learningcited by 45
  7. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  8. 2021Counterfactual Explanations in Sequential Decision Making Under Uncertaintycited by 37
  9. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  10. 2022Counterfactual harmcited by 29
  11. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  12. 2023Causal Reinforcement Learning: A Surveycited by 23
38 more
  1. 2021Counterfactual Temporal Point Processescited by 20
  2. 2021Dynamic Causal Bayesian Optimizationcited by 17
  3. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  4. 2021Causal Multi-Agent Reinforcement Learning: Review and Open Problemscited by 16
  5. 2021Learning Generalized Gumbel-max Causal Mechanismscited by 16
  6. 2022On the link between conscious function and general intelligence in humans and machinescited by 16
  7. 2021Directive Explanations for Actionable Explainability in Machine Learning Applicationscited by 14
  8. 2023Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Modelcited by 14
  9. 2024Counterfactual Token Generation in Large Language Modelscited by 14
  10. 2023Finding Counterfactually Optimal Action Sequences in Continuous State Spacescited by 13
  11. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 13
  12. 2022Adversarial Counterfactual Environment Model Learningcited by 12
  13. 2024Causal Action Influence Aware Counterfactual Data Augmentationcited by 12
  14. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learningcited by 11
  15. 2022Counterfactual Analysis in Dynamic Latent State Modelscited by 10
  16. 2025ExoPredicator: Learning Abstract Models of Dynamic Worlds for Robot Planningcited by 9
  17. 2022Actual Causality and Responsibility Attribution in Decentralized Partially Observable Markov Decision Processescited by 8
  18. 2026CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learningcited by 8
  19. 2022Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialoguescited by 7
  20. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  21. 2025Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learningcited by 6
  22. 2022Counterfactuals for the Futurecited by 5
  23. 2024Counterfactual Influence in Markov Decision Processescited by 5
  24. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  25. 2022Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environmentscited by 4
  26. 2023Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learningcited by 4
  27. 2024Automated Discovery of Functional Actual Causes in Complex Environmentscited by 4
  28. 2026CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluationcited by 4
  29. 2021Learning Under Adversarial and Interventional Shiftscited by 3
  30. 2021Accelerating the Learning of TAMER with Counterfactual Explanationscited by 3
  31. 2022Causal Temporal Reasoning for Markov Decision Processescited by 3
  32. 2023Towards Computationally Efficient Responsibility Attribution in Decentralized Partially Observable MDPscited by 3
  33. 2023Where and How to Attack? A Causality-Inspired Recipe for Generating Counterfactual Adversarial Examplescited by 3
  34. 2024What Hides behind Unfairness? Exploring Dynamics Fairness in Reinforcement Learningcited by 3
  35. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
  36. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  37. 2025Robust Counterfactual Inference in Markov Decision Processescited by 3
  38. 2021Did I do that? Blame as a means to identify controlled effects in reinforcement learningcited by 2

50 most cited of 82 All 82 (JSON)