# Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps

arXiv: https://arxiv.org/abs/2005.08874 · HTML: https://mlsanity.com/paper/2005.08874/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2020.

## Built on by (most cited first, 30 of 30)

- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations](https://mlsanity.com/paper/2302.12689/) (2023)
- [Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes](https://mlsanity.com/paper/2312.11118/) (2023)
- [Are we measuring trust correctly in explainability, interpretability, and transparency research?](https://mlsanity.com/paper/2209.00651/) (2022)
- [Explainable Reinforcement Learning via Model Transforms](https://mlsanity.com/paper/2209.12006/) (2022)
- [Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents](https://mlsanity.com/paper/2210.11825/) (2022)
- ["I Don't Think So": Summarizing Policy Disagreements for Agent Comparison](https://mlsanity.com/paper/2102.03064/) (2021)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [RACCER: Towards Reachable and Certain Counterfactual Explanations for Reinforcement Learning](https://mlsanity.com/paper/2303.04475/) (2023)
- [Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning](https://mlsanity.com/paper/2409.03632/) (2024)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents](https://mlsanity.com/paper/2101.07312/) (2021)
- [Exploring The Role of Local and Global Explanations in Recommender Systems](https://mlsanity.com/paper/2109.13301/) (2021)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [An Information Bottleneck Characterization of the Understanding-Workload Tradeoff](https://mlsanity.com/paper/2310.07802/) (2023)
- [ADESSE: Advice Explanations in Complex Repeated Decision-Making Environments](https://mlsanity.com/paper/2405.20705/) (2024)
- [Towards Balancing Preference and Performance through Adaptive Personalized Explainability](https://mlsanity.com/paper/2504.13856/) (2025)
- [Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour](https://mlsanity.com/paper/2505.17801/) (2025)
- [Behaviour Discovery and Attribution for Explainable Reinforcement Learning](https://mlsanity.com/paper/2503.14973/) (2025)
- [Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2504.05393/) (2025)
- [Neural Distillation as a State Representation Bottleneck in Reinforcement Learning](https://mlsanity.com/paper/2210.02224/) (2022)
- [(When) Are Contrastive Explanations of Reinforcement Learning Helpful?](https://mlsanity.com/paper/2211.07719/) (2022)
- [Measuring Perceived Trust in XAI-Assisted Decision-Making by Eliciting a Mental Model](https://mlsanity.com/paper/2307.11765/) (2023)
- [Reconciling Privacy and Explainability in High-Stakes: A Systematic Inquiry](https://mlsanity.com/paper/2412.20798/) (2024)
- [Interpretable ML for Imbalanced Data](https://mlsanity.com/paper/2212.07743/) (2022)
- [ASQ-IT: Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2301.09941/) (2023)
- [BET: Explaining Deep Reinforcement Learning through The Error-Prone Decisions](https://mlsanity.com/paper/2401.07263/) (2024)
- [Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learning](https://mlsanity.com/paper/2511.20591/) (2025)
- [Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2606.24622/) (2026)

## Abstract (arXiv)

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as the environments in which they act have large state spaces, and their decision-making can be affected by delayed rewards, making it difficult to analyze their behavior. To address this problem, several approaches have been developed. Some approaches attempt to convey the textitglobal behavior of the agent, describing the actions it takes in different states. Other approaches devised textitlocal explanations which provide information regarding the agent's decision-making in a particular state. In this paper, we combine global and local explanation methods, and evaluate their joint and separate contributions, providing (to the best of our knowledge) the first user study of combined local and global explanations for RL agents. Specifically, we augment strategy summaries that extract important trajectories of states from simulations of the agent with saliency maps which show what information the agent attends to. Our results show that the choice of what states to include in the summary (global information) strongly affects people's understanding of agents: participants shown summaries that included important states significantly outperformed participants who were presented with agent behavior in a randomly set of chosen world-states. We find mixed results with respect to augmenting demonstrations with saliency maps (local information), as the addition of saliency maps did not significantly improve performance in most cases. However, we do find some evidence that saliency maps can help users better understand what information the agent relies on in its decision making, suggesting avenues for future work that can further improve explanations of RL agents.