Explaining Reinforcement Learning to Mere Mortals: An Empirical Study
Not on the map. Why?
Abstract (arXiv)
We present a user study to investigate the impact of explanations on non-experts' understanding of reinforcement learning (RL) agents. We investigate both a common RL visualization, saliency maps (the focus of attention), and a more recent explanation type, reward-decomposition bars (predictions of future types of rewards). We designed a 124 participant, four-treatment experiment to compare participants' mental models of an RL agent in a simple Real-Time Strategy (RTS) game. Our results show that the combination of both saliency and reward bars were needed to achieve a statistically significant improvement in mental model score over the control. In addition, our qualitative analysis of the data reveals a number of effects for further study.
Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Mar 2019.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 3; 2022: 7; 2023: 3; 2024: 5; 2025: 1; 2026: 2.
21 papers.
Sort:
By area:
- 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
- 2021Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methodscited by 31
- 2022A Survey of Explainable Reinforcement Learningcited by 31
- 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
- 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agentscited by 10
- 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
- 2022Explaining Reinforcement Learning Policies through Counterfactual Trajectoriescited by 7
- 2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Drivingcited by 7
- 2022Interpretable Learned Emergent Communication for Human-Agent Teamscited by 6
- 2022Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applicationscited by 6
- 2022Experiential Explanations for Reinforcement Learningcited by 5
- 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agentscited by 4
9 more
- 2021Explaining Reward Functions to Humans for Better Human-Robot Collaborationcited by 3
- 2023An Information Bottleneck Characterization of the Understanding-Workload Tradeoffcited by 3
- 2024SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Taskscited by 3
- 2025Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policiescited by 1
- 2024Experiments with Encoding Structured Data for Neural Networkscited by 0
- 2024How to Measure Human-AI Prediction Accuracy in Explainable AI Systemscited by 0
- 2024Effective Reward Specification in Deep Reinforcement Learningcited by 0
- 2026Explaining Reinforcement Learning Agents via Inductive Logic Programmingcited by 0
- 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agentscited by 0
- 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents18 Aug 2026
- 2026Explaining Reinforcement Learning Agents via Inductive Logic Programming15 Jul 2026
- 2025Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies20 Jul 2025
- 2024Effective Reward Specification in Deep Reinforcement Learning10 Dec 2024
- 2024SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Tasks19 Nov 2024
- 2024How to Measure Human-AI Prediction Accuracy in Explainable AI Systems23 Aug 2024
- 2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Driving18 Mar 2024
- 2024Experiments with Encoding Structured Data for Neural Networks15 Feb 2024
- 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes18 Dec 2023
- 2023An Information Bottleneck Characterization of the Understanding-Workload Tradeoff11 Oct 2023
- 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples11 Oct 2023
- 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents21 Oct 2022
9 more
- 2022Experiential Explanations for Reinforcement Learning10 Oct 2022
- 2022Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applications5 Jul 2022
- 2022A Survey of Explainable Reinforcement Learning17 Feb 2022
- 2022Explaining Reinforcement Learning Policies through Counterfactual Trajectories29 Jan 2022
- 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI20 Jan 2022
- 2022Interpretable Learned Emergent Communication for Human-Agent Teams19 Jan 2022
- 2021Explaining Reward Functions to Humans for Better Human-Robot Collaboration8 Oct 2021
- 2021Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods25 Feb 2021
- 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents18 Jan 2021
- 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
- 2021Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methodscited by 31
- 2022A Survey of Explainable Reinforcement Learningcited by 31
- 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
- 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agentscited by 10
- 2022Explaining Reinforcement Learning Policies through Counterfactual Trajectoriescited by 7
- 2022Interpretable Learned Emergent Communication for Human-Agent Teamscited by 6
- 2022Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applicationscited by 6
- 2022Experiential Explanations for Reinforcement Learningcited by 5
- 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agentscited by 4
- 2021Explaining Reward Functions to Humans for Better Human-Robot Collaborationcited by 3
- 2023An Information Bottleneck Characterization of the Understanding-Workload Tradeoffcited by 3
12 most cited of 18 All 18 (JSON)
- 2026Explaining Reinforcement Learning Agents via Inductive Logic Programming15 Jul 2026
- 2025Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies20 Jul 2025
- 2024Effective Reward Specification in Deep Reinforcement Learning10 Dec 2024
- 2024SkillTree: Explainable Skill-Based Deep Reinforcement Learning for Long-Horizon Control Tasks19 Nov 2024
- 2024How to Measure Human-AI Prediction Accuracy in Explainable AI Systems23 Aug 2024
- 2024Experiments with Encoding Structured Data for Neural Networks15 Feb 2024
- 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes18 Dec 2023
- 2023An Information Bottleneck Characterization of the Understanding-Workload Tradeoff11 Oct 2023
- 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents21 Oct 2022
- 2022Experiential Explanations for Reinforcement Learning10 Oct 2022
- 2022Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applications5 Jul 2022
- 2022A Survey of Explainable Reinforcement Learning17 Feb 2022
12 newest of 18 All 18 (JSON)