# Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanations

arXiv: https://arxiv.org/abs/2109.13978 · HTML: https://mlsanity.com/paper/2109.13978/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 12 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 6 of 6)

- [Visualizing and Understanding Atari Agents](https://mlsanity.com/paper/1711.00138/) (2017)
- [Towards Interpretable Reinforcement Learning Using Attention Augmented Agents](https://mlsanity.com/paper/1906.02500/) (2019)
- [Establishing Appropriate Trust via Critical States](https://mlsanity.com/paper/1810.08174/) (2018)
- [Transparency and Explanation in Deep Reinforcement Learning Neural Networks](https://mlsanity.com/paper/1809.06061/) (2018)
- [Learning Finite State Representations of Recurrent Policy Networks](https://mlsanity.com/paper/1811.12530/) (2018)
- [Counterfactual States for Atari Agents via Generative Deep Learning](https://mlsanity.com/paper/1909.12969/) (2019)

## Built on by (most cited first, 2 of 2)

- [Structure and Reduction of MCTS for Explainable-AI](https://mlsanity.com/paper/2408.05488/) (2024)
- [Beyond Value: CHECKLIST for Testing Inferences in Planning-Based RL](https://mlsanity.com/paper/2206.02039/) (2022)

## Abstract (arXiv)

Enabling humans to identify potential flaws in an agent's decision making is an important Explainable AI application. We consider identifying such flaws in a planning-based deep reinforcement learning (RL) agent for a complex real-time strategy game. In particular, the agent makes decisions via tree search using a learned model and evaluation function over interpretable states and actions. This gives the potential for humans to identify flaws at the level of reasoning steps in the tree, even if the entire reasoning process is too complex to understand. However, it is unclear whether humans will be able to identify such flaws due to the size and complexity of trees. We describe a user interface and case study, where a small group of AI experts and developers attempt to identify reasoning flaws due to inaccurate agent learning. Overall, the interface allowed the group to identify a number of significant flaws of varying types, demonstrating the promise of this approach.