mlsanityData to 7 Oct 2026

Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities

arXiv 2210.11846 ↗First posted 21 Oct 2022cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

While AI algorithms have shown remarkable success in various fields, their lack of transparency hinders their application to real-life tasks. Although explanations targeted at non-experts are necessary for user trust and human-AI collaboration, the majority of explanation methods for AI are focused on developers and expert users. Counterfactual explanations are local explanations that offer users advice on what can be changed in the input for the output of the black-box model to change. Counterfactuals are user-friendly and provide actionable advice for achieving the desired output from the AI system. While extensively researched in supervised learning, there are few methods applying them to reinforcement learning (RL). In this work, we explore the reasons for the underrepresentation of a powerful explanation method in RL. We start by reviewing the current work in counterfactual explanations in supervised learning. Additionally, we explore the differences between counterfactual explanations in supervised learning and RL and identify the main challenges that prevent the adoption of methods from supervised in reinforcement learning. Finally, we redefine counterfactuals for RL and propose research directions for implementing counterfactuals in RL.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 59 of its 106 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

59 of 106 references.

References · 59

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2016Concrete Problems in AI Safetycited by 1,582
  8. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  9. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  10. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  11. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  12. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
38 more
  1. 2018Recognition in Terra Incognitacited by 655
  2. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 612
  3. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  4. 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 343
  5. 2018Actionable Recourse in Linear Classificationcited by 327
  6. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  7. 2019Challenges of Real-World Reinforcement Learningcited by 282
  8. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  9. 2020Algorithmic Recourse: from Counterfactual Explanations to Interventionscited by 250
  10. 2019FACE: Feasible and Actionable Counterfactual Explanationscited by 232
  11. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  12. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  13. 2019Interpretable Counterfactual Explanations Guided by Prototypescited by 223
  14. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  15. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  16. 2020Multi-Objective Counterfactual Explanationscited by 157
  17. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  18. 2019Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifierscited by 143
  19. 2017Visualizing and Understanding Atari Agentscited by 141
  20. 2020Explainable Reinforcement Learning: A Surveycited by 139
  21. 2020Algorithmic recourse under imperfect causal knowledge: a probabilistic approachcited by 139
  22. 2019Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systemscited by 128
  23. 2020A survey of algorithmic recourse: definitions, formulations, solutions, and prospectscited by 118
  24. 2020Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehiclescited by 101
  25. 2020Explainability in Deep Reinforcement Learningcited by 96
  26. 2019Hindsight Credit Assignmentcited by 74
  27. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  28. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  29. 2020Quantifying Differences in Reward Functionscited by 62
  30. 2017Interpretable & Explorable Approximations of Black Box Modelscited by 54
  31. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  32. 2019Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attributioncited by 50
  33. 2017Inverse Classification for Comparison-based Interpretability in Machine Learningcited by 49
  34. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  35. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  36. 2019Generation of Policy-Level Explanations for Reinforcement Learningcited by 37
  37. 2019CERTIFAI: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence modelscited by 34
  38. 2019Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulationcited by 33

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 3.

12 papers.

  1. 2024Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimizationcited by 4
  2. 2025Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviourcited by 3
  3. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Modelscited by 2
  4. 2024Explaining Learned Reward Functions with Counterfactual Trajectoriescited by 1
  5. 2024Semifactual Explanations for Reinforcement Learningcited by 1
  6. 2025Explainable Reinforcement Learning Agents Using World Modelscited by 1
  7. 2025Counterfactual Strategies for Markov Decision Processescited by 1
  8. 2025Counterfactual Explanations for Continuous Action Reinforcement Learningcited by 1
  9. 2025SHAP-Guided Kernel Actor-Critic for Explainable Reinforcement Learningcited by 0
  10. 2026SocialFiVis: A Visual Analytics Sandbox for LLM-Grounded Multi-Agent Simulation in Social Financecited by 0
  11. 2026Explaining Reinforcement Learning Decisions in Self-adaptive Systemscited by 0
  12. 2026Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learningcited by 0