mlsanityData to 7 Oct 2026

Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution

arXiv 1912.12191 ↗First posted 23 Dec 2019cs.CV cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents. Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action. Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent. Our proposed approach, SARFA (Specific and Relevant Feature Attribution), generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency. The first captures the impact of perturbation on the relative expected reward of the action to be explained. The second downweighs irrelevant features that alter the relative expected rewards of actions other than the action to be explained. We compare SARFA with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders). We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches. For the code release and demo videos, see https://nikaashpuri.github.io/sarfa-saliency/.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 12; 2023: 8; 2024: 8; 2025: 11; 2026: 5.

50 papers.

Sort:

By area:

  1. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  2. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  3. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  4. 2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 41
  5. 2021Counterfactual Explanations in Sequential Decision Making Under Uncertaintycited by 36
  6. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  7. 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
  8. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
  9. 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 12
  10. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  11. 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agentscited by 10
  12. 2024Understanding Individual Agent Importance in Multi-Agent System via Counterfactual Reasoningcited by 10
38 more
  1. 2025Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMscited by 8
  2. 2022Explaining Reinforcement Learning Policies through Counterfactual Trajectoriescited by 7
  3. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
  4. 2023Explaining RL Decisions with Trajectoriescited by 7
  5. 2024RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanationcited by 7
  6. 2021Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learningcited by 6
  7. 2024How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency mapscited by 6
  8. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 5
  9. 2023PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Marketscited by 5
  10. 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agentscited by 4
  11. 2022ReCCoVER: Detecting Causal Confusion for Explainable Reinforcement Learningcited by 4
  12. 2022Human-AI Shared Control via Policy Dissectioncited by 4
  13. 2024ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policiescited by 4
  14. 2022Policy Distillation with Selective Input Gradient Regularization for Efficient Interpretabilitycited by 2
  15. 2023Counterfactual Explanation Policies in RLcited by 2
  16. 2024XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniquescited by 2
  17. 2025Behaviour Discovery and Attribution for Explainable Reinforcement Learningcited by 2
  18. 2025Towards Piece-by-Piece Explanations for Chess Positions with SHAPcited by 2
  19. 2022Neural Distillation as a State Representation Bottleneck in Reinforcement Learningcited by 1
  20. 2022Causal Explanation for Reinforcement Learning: Quantifying State and Temporal Importancecited by 1
  21. 2024'Explaining RL Decisions with Trajectories': A Reproducibility Studycited by 1
  22. 2025A Comparative User Evaluation of XRL Explanations using Goal Identificationcited by 1
  23. 2025Approximating Shapley Explanations in Reinforcement Learningcited by 1
  24. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  25. 2026BXRL: Behavior-Explainable Reinforcement Learningcited by 1
  26. 2021The Atari Data Scrapercited by 0
  27. 2022Interpretable Deep Reinforcement Learning for Green Security Games with Real-Time Informationcited by 0
  28. 2023CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systemscited by 0
  29. 2024Revealing the Learning Process in Reinforcement Learning Agents Through Attention-Oriented Metricscited by 0
  30. 2024Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learningcited by 0
  31. 2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0
  32. 2025Pay Attention to What and Where? Interpretable Feature Extractor in Vision-based Deep Reinforcement Learningcited by 0
  33. 2025MAGIC-MASK: Multi-Agent Guided Inter-Agent Collaboration with Mask-Based Explainability for Reinforcement Learningcited by 0
  34. 2025Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learningcited by 0
  35. 2026Reward-Conditioned Attention: How Reward Design Shapes What Autonomous Driving Agents Seecited by 0
  36. 2026Communicating Chess Strategies in Natural Languagecited by 0
  37. 2026ConceptTree: Bringing Semantic Transparency to Black-Box Decision Making for Robotic Manipulationcited by 0
  38. 2026Explaining Reinforcement Learning Decisions in Self-adaptive Systemscited by 0