mlsanityData to 7 Oct 2026

A Survey on Causal Reinforcement Learning

arXiv 2302.05209 ↗First posted 10 Feb 2023cs.AI cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

While Reinforcement Learning (RL) achieves tremendous success in sequential decision-making problems of many domains, it still faces key challenges of data inefficiency and the lack of interpretability. Interestingly, many researchers have leveraged insights from the causality literature recently, bringing forth flourishing works to unify the merits of causality and address well the challenges from RL. As such, it is of great necessity and significance to collate these Causal Reinforcement Learning (CRL) works, offer a review of CRL methods, and investigate the potential functionality from causality toward RL. In particular, we divide existing CRL approaches into two categories according to whether their causality-based information is given in advance or not. We further analyze each category in terms of the formalization of different models, ranging from the Markov Decision Process (MDP), Partially Observed Markov Decision Process (POMDP), Multi-Arm Bandits (MAB), and Dynamic Treatment Regime (DTR). Moreover, we summarize the evaluation matrices and open sources while we discuss emerging applications, along with promising prospects for the future development of CRL.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 119 of its 234 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

119 of 234 references.

References · 119

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
38 more
  1. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  2. 2017Deep Reinforcement Learning that Matterscited by 727
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  4. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 475
  5. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  6. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  9. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  10. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
  11. 2016Learning Representations for Counterfactual Inferencecited by 327
  12. 2017Deep Reinforcement Learning: An Overviewcited by 327
  13. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  14. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  15. 2019Causal Confusion in Imitation Learningcited by 286
  16. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  17. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  18. 2020A Survey on Causal Inferencecited by 202
  19. 2013Learning Bayesian Networks: The Combination of Knowledge and Statistical Datacited by 185
  20. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  21. 2019Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changescited by 166
  22. 2013Causal Inference in the Presence of Latent Variables and Selection Biascited by 165
  23. 2017Neural Episodic Controlcited by 148
  24. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 138
  25. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  26. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  27. 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
  28. 2022Causal Machine Learning: A Survey and Open Problemscited by 107
  29. 2020Invariant Causal Prediction for Block MDPscited by 104
  30. 2016Causal Bandits: Learning Good Interventions via Causal Inferencecited by 100
  31. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  32. 2020High-recall causal discovery for autocorrelated time series with latent confounderscited by 82
  33. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 81
  34. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  35. 2018Confounding-Robust Policy Improvementcited by 73
  36. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 70
  37. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 70
  38. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65

50 most cited of 119 All 119 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 12; 2025: 15; 2026: 11.

43 papers.

Sort:

By area:

  1. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  2. 2023Neurosymbolic Reinforcement Learning and Planning: A Surveycited by 24
  3. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  4. 2024ADAM: An Embodied Causal Agent in Open-World Environmentscited by 13
  5. 2025PHASE-Net: Physics-Grounded Harmonic Attention System for Efficient Remote Photoplethysmography Measurementcited by 10
  6. 2025ExoPredicator: Learning Abstract Models of Dynamic Worlds for Robot Planningcited by 8
  7. 2024Towards Identifiable Latent Additive Noise Modelscited by 5
  8. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  9. 2024Counterfactual Influence in Markov Decision Processescited by 4
  10. 2024Variable-Agnostic Causal Exploration for Reinforcement Learningcited by 4
  11. 2023Emergence and Causality in Complex Systems: A Survey on Causal Emergence and Related Quantitative Studiescited by 3
  12. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
31 more
  1. 2024Causal Reasoning in Software Quality Assurance: A Systematic Reviewcited by 3
  2. 2025Causal Knowledge Transfer for Multi-Agent Reinforcement Learning in Dynamic Environmentscited by 3
  3. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learningcited by 3
  4. 2023Causal State Distillation for Explainable Reinforcement Learningcited by 2
  5. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flowscited by 2
  6. 2024Systems with Switching Causal Relations: A Meta-Causal Perspectivecited by 2
  7. 2025A Causal Lens for Learning Long-term Fair Policiescited by 2
  8. 2025Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlationscited by 2
  9. 2023Timing Process Interventions with Causal Inference and Reinforcement Learningcited by 1
  10. 2024Preparing for Black Swans: The Antifragility Imperative for Machine Learningcited by 1
  11. 2025Quantifying First-Order Markov Violations in Noisy Reinforcement Learning: A Causal Discovery Approachcited by 1
  12. 2025Towards Causal Model-Based Policy Optimizationcited by 1
  13. 2025A Roadmap Towards Improving Multi-Agent Reinforcement Learning With Causal Discovery And Inferencecited by 1
  14. 2025Causal-Paced Deep Reinforcement Learningcited by 1
  15. 2025DODO: Causal Structure Learning with Budgeted Interventionscited by 1
  16. 2025Improving Robustness to Out-of-Distribution States in Imitation Learning via Deep Koopman-Boosted Diffusion Policycited by 1
  17. 2025LacaDM: A Latent Causal Diffusion Model for Multiobjective Reinforcement Learningcited by 1
  18. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  19. 2024CIER: A Novel Experience Replay Approach with Causal Inference in Deep Reinforcement Learningcited by 0
  20. 2024Wind Estimation in Unmanned Aerial Vehicles with Causal Machine Learningcited by 0
  21. 2025Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challengescited by 0
  22. 2025Causal-Aware Intelligent QoE Optimization for VR Interaction with Adaptive Keyframe Extractioncited by 0
  23. 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learningcited by 0
  24. 2026CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategiescited by 0
  25. 2026Teleological Inference in Structural Causal Models via Intentional Interventionscited by 0
  26. 2026A Closer Look at the Application of Causal Inference in Graph Representation Learningcited by 0
  27. 2026Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmarkcited by 0
  28. 2026Property-driven Causal Abstractions for Markov Decision Processescited by 0
  29. 2026Temporal-Causal Inference for Reinforcement Learning via Automata Learningcited by 0
  30. 2026Causal Retention in Interactive Agents: Interface Factorization and Selective Adaptationcited by 0
  31. 2026FLOW: Feature-Level Optimal Warping for Generalized Remote Physiological Measurementcited by 0