# RACCER: Towards Reachable and Certain Counterfactual Explanations for Reinforcement Learning

arXiv: https://arxiv.org/abs/2303.04475 · HTML: https://mlsanity.com/paper/2303.04475/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 26 of 26)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Actionable Recourse in Linear Classification](https://mlsanity.com/paper/1809.06514/) (2018)
- [Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives](https://mlsanity.com/paper/1802.07623/) (2018)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [FACE: Feasible and Actionable Counterfactual Explanations](https://mlsanity.com/paper/1909.09369/) (2019)
- [Interpretable Counterfactual Explanations Guided by Prototypes](https://mlsanity.com/paper/1907.02584/) (2019)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [Multi-Objective Counterfactual Explanations](https://mlsanity.com/paper/2004.11165/) (2020)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [Algorithmic recourse under imperfect causal knowledge: a probabilistic approach](https://mlsanity.com/paper/2006.06831/) (2020)
- [A survey of algorithmic recourse: definitions, formulations, solutions, and prospects](https://mlsanity.com/paper/2010.04050/) (2020)
- [Inverse Classification for Comparison-based Interpretability in Machine Learning](https://mlsanity.com/paper/1712.08443/) (2017)
- [Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps](https://mlsanity.com/paper/2005.08874/) (2020)
- [Counterfactual Explanations for Machine Learning: Challenges Revisited](https://mlsanity.com/paper/2106.07756/) (2021)
- [GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations](https://mlsanity.com/paper/2302.12689/) (2023)
- [Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutions](https://mlsanity.com/paper/2107.09734/) (2021)
- [Model-agnostic and Scalable Counterfactual Explanations via Reinforcement Learning](https://mlsanity.com/paper/2106.02597/) (2021)
- [Counterfactual States for Atari Agents via Generative Deep Learning](https://mlsanity.com/paper/1909.12969/) (2019)
- [Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning?](https://mlsanity.com/paper/2012.01281/) (2020)

## Built on by (most cited first, 6 of 6)

- [ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policies](https://mlsanity.com/paper/2402.06503/) (2024)
- [Semifactual Explanations for Reinforcement Learning](https://mlsanity.com/paper/2409.05435/) (2024)
- [Explainable Reinforcement Learning Agents Using World Models](https://mlsanity.com/paper/2505.08073/) (2025)
- [Personalized Path Recourse for Reinforcement Learning Agents](https://mlsanity.com/paper/2312.08724/) (2023)
- [Explaining Reinforcement Learning Decisions in Self-adaptive Systems](https://mlsanity.com/paper/2608.14620/) (2026)
- [Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2608.14963/) (2026)

## Abstract (arXiv)

While reinforcement learning (RL) algorithms have been successfully applied to numerous tasks, their reliance on neural networks makes their behavior difficult to understand and trust. Counterfactual explanations are human-friendly explanations that offer users actionable advice on how to alter the model inputs to achieve the desired output from a black-box system. However, current approaches to generating counterfactuals in RL ignore the stochastic and sequential nature of RL tasks and can produce counterfactuals that are difficult to obtain or do not deliver the desired outcome. In this work, we propose RACCER, the first RL-specific approach to generating counterfactual explanations for the behavior of RL agents. We first propose and implement a set of RL-specific counterfactual properties that ensure easily reachable counterfactuals with highly probable desired outcomes. We use a heuristic tree search of the agent's execution trajectories to find the most suitable counterfactuals based on the defined properties. We evaluate RACCER in two tasks as well as conduct a user study to show that RL-specific counterfactuals help users better understand agents' behavior compared to the current state-of-the-art approaches.