# Explaining Reinforcement Learning to Mere Mortals: An Empirical Study

arXiv: https://arxiv.org/abs/1903.09708 · HTML: https://mlsanity.com/paper/1903.09708/

## Key facts

- Cited by 21 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 22 Mar 2019.

## Built on by (most cited first, 21 of 21)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes](https://mlsanity.com/paper/2312.11118/) (2023)
- [Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents](https://mlsanity.com/paper/2210.11825/) (2022)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Explaining Reinforcement Learning Policies through Counterfactual Trajectories](https://mlsanity.com/paper/2201.12462/) (2022)
- [Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Driving](https://mlsanity.com/paper/2403.12176/) (2024)
- [Interpretable Learned Emergent Communication for Human-Agent Teams](https://mlsanity.com/paper/2201.07452/) (2022)
- [Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applications](https://mlsanity.com/paper/2207.01911/) (2022)
- [Experiential Explanations for Reinforcement Learning](https://mlsanity.com/paper/2210.04723/) (2022)
- [Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents](https://mlsanity.com/paper/2101.07312/) (2021)
- [Explaining Reward Functions to Humans for Better Human-Robot Collaboration](https://mlsanity.com/paper/2110.04192/) (2021)
- [An Information Bottleneck Characterization of the Understanding-Workload Tradeoff](https://mlsanity.com/paper/2310.07802/) (2023)
- [SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Tasks](https://mlsanity.com/paper/2411.12173/) (2024)
- [Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies](https://mlsanity.com/paper/2507.14901/) (2025)
- [Experiments with Encoding Structured Data for Neural Networks](https://mlsanity.com/paper/2402.10290/) (2024)
- [How to Measure Human-AI Prediction Accuracy in Explainable AI Systems](https://mlsanity.com/paper/2409.00069/) (2024)
- [Effective Reward Specification in Deep Reinforcement Learning](https://mlsanity.com/paper/2412.07177/) (2024)
- [Explaining Reinforcement Learning Agents via Inductive Logic Programming](https://mlsanity.com/paper/2607.13655/) (2026)
- [Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents](https://mlsanity.com/paper/2608.17524/) (2026)

## Abstract (arXiv)

We present a user study to investigate the impact of explanations on non-experts' understanding of reinforcement learning (RL) agents. We investigate both a common RL visualization, saliency maps (the focus of attention), and a more recent explanation type, reward-decomposition bars (predictions of future types of rewards). We designed a 124 participant, four-treatment experiment to compare participants' mental models of an RL agent in a simple Real-Time Strategy (RTS) game. Our results show that the combination of both saliency and reward bars were needed to achieve a statistically significant improvement in mental model score over the control. In addition, our qualitative analysis of the data reveals a number of effects for further study.