# SAFE-RL: Saliency-Aware Counterfactual Explainer for Deep Reinforcement Learning Policies

arXiv: https://arxiv.org/abs/2404.18326 · HTML: https://mlsanity.com/paper/2404.18326/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 28 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Eigen-CAM: Class Activation Map using Principal Components](https://mlsanity.com/paper/2008.00299/) (2020)
- [Diffusion Visual Counterfactual Explanations](https://mlsanity.com/paper/2210.11841/) (2022)
- [Diffusion Models for Counterfactual Explanations](https://mlsanity.com/paper/2203.15636/) (2022)
- [AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networks](https://mlsanity.com/paper/1911.11897/) (2019)
- [STEEX: Steering Counterfactual Explanations with Semantics](https://mlsanity.com/paper/2111.09094/) (2021)
- [Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning](https://mlsanity.com/paper/2101.12446/) (2021)
- [Cycle-Consistent Counterfactuals by Latent Transformations](https://mlsanity.com/paper/2203.15064/) (2022)
- [OCTET: Object-aware Counterfactual Explanations](https://mlsanity.com/paper/2211.12380/) (2022)
- [GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations](https://mlsanity.com/paper/2302.12689/) (2023)
- [FASTER-CE: Fast, Sparse, Transparent, and Robust Counterfactual Explanations](https://mlsanity.com/paper/2210.06578/) (2022)
- [SAFE: Saliency-Aware Counterfactual Explanations for DNN-based Automated Driving Systems](https://mlsanity.com/paper/2307.15786/) (2023)
- [Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation](https://mlsanity.com/paper/2305.16532/) (2023)

## Built on by (most cited first, 8 of 8)

- [Good Data Is All Imitation Learning Needs](https://mlsanity.com/paper/2409.17605/) (2024)
- [From Actions to Words: Towards Abstractive-Textual Policy Summarization in RL](https://mlsanity.com/paper/2503.10509/) (2025)
- [Realistic Counterfactual Explanations for Machine Learning-Controlled Mobile Robots using 2D LiDAR](https://mlsanity.com/paper/2505.06906/) (2025)
- [Position: Intelligent Coding Systems Should Write Programs with Justifications](https://mlsanity.com/paper/2508.06017/) (2025)
- [COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenance](https://mlsanity.com/paper/2603.07546/) (2026)
- [Engineering Trustworthy Agentic AI for Critical Systems](https://mlsanity.com/paper/2607.18548/) (2026)
- [Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2608.14963/) (2026)
- [Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle](https://mlsanity.com/paper/2608.30581/) (2026)

## Abstract (arXiv)

While Deep Reinforcement Learning (DRL) has emerged as a promising solution for intricate control tasks, the lack of explainability of the learned policies impedes its uptake in safety-critical applications, such as automated driving systems (ADS). Counterfactual (CF) explanations have recently gained prominence for their ability to interpret black-box Deep Learning (DL) models. CF examples are associated with minimal changes in the input, resulting in a complementary output by the DL model. Finding such alternations, particularly for high-dimensional visual inputs, poses significant challenges. Besides, the temporal dependency introduced by the reliance of the DRL agent action on a history of past state observations further complicates the generation of CF examples. To address these challenges, we propose using a saliency map to identify the most influential input pixels across the sequence of past observed states by the agent. Then, we feed this map to a deep generative model, enabling the generation of plausible CFs with constrained modifications centred on the salient regions. We evaluate the effectiveness of our framework in diverse domains, including ADS, Atari Pong, Pacman and space-invaders games, using traditional performance metrics such as validity, proximity and sparsity. Experimental results demonstrate that this framework generates more informative and plausible CFs than the state-of-the-art for a wide range of environments and DRL agents. In order to foster research in this area, we have made our datasets and codes publicly available at https://github.com/Amir-Samadi/SAFE-RL.