# Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program

arXiv: https://arxiv.org/abs/2106.05506 · HTML: https://mlsanity.com/paper/2106.05506/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 7 of 7)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Recurrent Neural Network Regularization](https://mlsanity.com/paper/1409.2329/) (2014)
- [Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning](https://mlsanity.com/paper/1912.05743/) (2019)

## Built on by (most cited first, 11 of 11)

- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [Capturing Humans' Mental Models of AI: An Item Response Theory Approach](https://mlsanity.com/paper/2305.09064/) (2023)
- [Incremental XAI: Memorable Understanding of AI with Incremental Explanations](https://mlsanity.com/paper/2404.06733/) (2024)
- [Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applications](https://mlsanity.com/paper/2207.01911/) (2022)
- [Military AI Needs Technically-Informed Regulation to Safeguard AI Research and its Applications](https://mlsanity.com/paper/2505.18371/) (2025)
- [Automating Governing Knowledge Commons and Contextual Integrity (GKC-CI) Privacy Policy Annotations with Large Language Models](https://mlsanity.com/paper/2311.02192/) (2023)
- [Assessing Trustworthiness of Autonomous Systems](https://mlsanity.com/paper/2305.03411/) (2023)
- [HEX: Human-in-the-loop Explainability via Deep Reinforcement Learning](https://mlsanity.com/paper/2206.01343/) (2022)
- [Taking the Intentional Stance Seriously, or "Intending" to Improve Cognitive Systems](https://mlsanity.com/paper/2209.11764/) (2022)
- [Gradient-Optimized Fuzzy Classifier: A Benchmark Study Against State-of-the-Art Models](https://mlsanity.com/paper/2504.16263/) (2025)
- [Object-Oriented Transition Modeling with Inductive Logic Programming](https://mlsanity.com/paper/2602.07602/) (2026)

## Abstract (arXiv)

The advances in artificial intelligence enabled by deep learning architectures are undeniable. In several cases, deep neural network driven models have surpassed human level performance in benchmark autonomy tasks. The underlying policies for these agents, however, are not easily interpretable. In fact, given their underlying deep models, it is impossible to directly understand the mapping from observations to actions for any reasonably complex agent. Producing this supporting technology to "open the black box" of these AI systems, while not sacrificing performance, was the fundamental goal of the DARPA XAI program. In our journey through this program, we have several "big picture" takeaways: 1) Explanations need to be highly tailored to their scenario; 2) many seemingly high performing RL agents are extremely brittle and are not amendable to explanation; 3) causal models allow for rich explanations, but how to present them isn't always straightforward; and 4) human subjects conjure fantastically wrong mental models for AIs, and these models are often hard to break. This paper discusses the origins of these takeaways, provides amplifying information, and suggestions for future work.