Data to 7 Oct 2026

Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning

arXiv 1912.05743 ↗First posted 9 Dec 2019cs.LG cs.AIcited by 60

Not on the map. Why?

Abstract (arXiv)

Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which they correspond to the semantics of RL environments. We use Atari games, a common benchmark for deep RL, to evaluate three types of saliency maps. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are best viewed as an exploratory tool rather than an explanatory tool.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 11; 2023: 11; 2024: 9; 2025: 11; 2026: 6.

60 papers.

Sort:

By area:

2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
2021A Survey on Interpretable Reinforcement Learningcited by 70
2022Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathologycited by 58
2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 41
2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 38
2021Counterfactual Explanations in Sequential Decision Making Under Uncertaintycited by 36
2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
2023Explainable Reinforcement Learning via a Causal World Modelcited by 22

38 more

2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
2024ViSaRL: Visual Reinforcement Learning Guided by Human Saliencycited by 12
2021Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Programcited by 11
2024Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Surveycited by 11
2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
2023Explainable Multi-Agent Reinforcement Learning for Temporal Queriescited by 9
2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Drivingcited by 7
2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
2021Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learningcited by 6
2022Explainability Via Causal Self-Talkcited by 6
2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
2021Explainable Deep Reinforcement Learning for Portfolio Management: An Empirical Approachcited by 5
2022Reinforcement Learning in Practice: Opportunities and Challengescited by 5
2024Behaviour Distillationcited by 5
2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agentscited by 4
2022Human-AI Shared Control via Policy Dissectioncited by 4
2024ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policiescited by 4
2023Learning Generalizable Agents via Saliency-Guided Features Decorrelationcited by 3
2023Towards interpretable-by-design deep learning algorithmscited by 3
2023On the Relationship Between Interpretability and Explainability in Machine Learningcited by 3
2024Exposing Image Classifier Shortcuts with Counterfactual Frequency (CoF) Tablescited by 3
2024Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Modelscited by 2
2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Modelscited by 2
2021An Approach to Partial Observability in Games: Learning to Both Act and Observecited by 1
2022Detecting danger in gridworlds using Gromov's Link Conditioncited by 1
2022Power and accountability in reinforcement learning applications to environmental policycited by 1
2023Colour versus Shape Goal Misgeneralization in Reinforcement Learning: A Case Studycited by 1
2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
2026BXRL: Behavior-Explainable Reinforcement Learningcited by 1
2021Abstraction, Validation, and Generalization for Explainable Artificial Intelligencecited by 0
2022System-Agnostic Meta-Learning for MDP-based Dynamic Scheduling via Descriptive Policycited by 0
2022Measuring Interventional Robustness in Reinforcement Learningcited by 0
2022Interpretable Deep Reinforcement Learning for Green Security Games with Real-Time Informationcited by 0
2023Explainable History Distillation by Marked Temporal Point Processcited by 0
2024Revealing the Learning Process in Reinforcement Learning Agents Through Attention-Oriented Metricscited by 0
2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0

50 most cited of 60 All 60 (JSON)