# Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2103.04067 · HTML: https://mlsanity.com/paper/2103.04067/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 6 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Visualizing and Understanding Atari Agents](https://mlsanity.com/paper/1711.00138/) (2017)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)
- [Towards Interpretable Reinforcement Learning Using Attention Augmented Agents](https://mlsanity.com/paper/1906.02500/) (2019)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [AGIL: Learning Attention from Human for Visuomotor Tasks](https://mlsanity.com/paper/1806.03960/) (2018)
- [Deep Learning for Video Game Playing](https://mlsanity.com/paper/1708.07902/) (2017)
- [Deep Attention Recurrent Q-Network](https://mlsanity.com/paper/1512.01693/) (2015)
- [Reinforcement Learning with Attention that Works: A Self-Supervised Approach](https://mlsanity.com/paper/1904.03367/) (2019)

## Built on by (most cited first, 9 of 9)

- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents](https://mlsanity.com/paper/2401.05821/) (2024)
- [Layer-Wise Relevance Propagation with Conservation Property for ResNet](https://mlsanity.com/paper/2407.09115/) (2024)
- [ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policies](https://mlsanity.com/paper/2402.06503/) (2024)
- [AEGIS: Human Attention-based Explainable Guidance for Intelligent Vehicle Systems](https://mlsanity.com/paper/2504.05950/) (2025)
- [Revealing Human Attention Patterns from Gameplay Analysis for Reinforcement Learning](https://mlsanity.com/paper/2504.11118/) (2025)
- [Interpretable Deep Reinforcement Learning for Green Security Games with Real-Time Information](https://mlsanity.com/paper/2211.04987/) (2022)
- [OPTIC-ER: A Reinforcement Learning Framework for Real-Time Emergency Response and Equitable Resource Allocation in Underserved African Communities](https://mlsanity.com/paper/2508.12943/) (2025)
- [Attention Lattice Adapter: Visual Explanation Generation for Visual Foundation Model](https://mlsanity.com/paper/2509.14664/) (2025)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has great potential for acquiring the optimal action in complex environments such as games and robot control. However, it is difficult to analyze the decision-making of the agent, i.e., the reasons it selects the action acquired by learning. In this work, we propose Mask-Attention A3C (Mask A3C), which introduces an attention mechanism into Asynchronous Advantage Actor-Critic (A3C), which is an actor-critic-based DRL method, and can analyze the decision-making of an agent in DRL. A3C consists of a feature extractor that extracts features from an image, a policy branch that outputs the policy, and a value branch that outputs the state value. In this method, we focus on the policy and value branches and introduce an attention mechanism into them. The attention mechanism applies a mask processing to the feature maps of each branch using mask-attention that expresses the judgment reason for the policy and state value with a heat map. We visualized mask-attention maps for games on the Atari 2600 and found we could easily analyze the reasons behind an agent's decision-making in various game tasks. Furthermore, experimental results showed that the agent could achieve a higher performance by introducing the attention mechanism.