# Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/1904.01318 · HTML: https://mlsanity.com/paper/1904.01318/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 15 of 15)

- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Investigating the Properties of Neural Network Representations in Reinforcement Learning](https://mlsanity.com/paper/2203.15955/) (2022)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Explaining Reinforcement Learning Policies through Counterfactual Trajectories](https://mlsanity.com/paper/2201.12462/) (2022)
- [Understanding the Evolution of Linear Regions in Deep Reinforcement Learning](https://mlsanity.com/paper/2210.13611/) (2022)
- [On the Safety of Interpretable Machine Learning: A Maximum Deviation Approach](https://mlsanity.com/paper/2211.01498/) (2022)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2504.05393/) (2025)
- [Introspection-based Explainable Reinforcement Learning in Episodic and Non-episodic Scenarios](https://mlsanity.com/paper/2211.12930/) (2022)
- [ASQ-IT: Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2301.09941/) (2023)
- [Action Q-Transformer: Visual Explanation in Deep Reinforcement Learning with Encoder-Decoder Model using Action Query](https://mlsanity.com/paper/2306.13879/) (2023)
- [Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models](https://mlsanity.com/paper/2605.14897/) (2026)

## Abstract (arXiv)

As deep reinforcement learning driven by visual perception becomes more widely used there is a growing need to better understand and probe the learned agents. Understanding the decision making process and its relationship to visual inputs can be very valuable to identify problems in learned behavior. However, this topic has been relatively under-explored in the research community. In this work we present a method for synthesizing visual inputs of interest for a trained agent. Such inputs or states could be situations in which specific actions are necessary. Further, critical states in which a very high or a very low reward can be achieved are often interesting to understand the situational awareness of the system as they can correspond to risky states. To this end, we learn a generative model over the state space of the environment and use its latent space to optimize a target function for the state of interest. In our experiments we show that this method can generate insights for a variety of environments and reinforcement learning methods. We explore results in the standard Atari benchmark games as well as in an autonomous driving simulator. Based on the efficiency with which we have been able to identify behavioural weaknesses with this technique, we believe this general approach could serve as an important tool for AI safety applications.