# Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recovery

arXiv: https://arxiv.org/abs/2101.01625 · HTML: https://mlsanity.com/paper/2101.01625/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 53 references that are in the map.
- First posted on arXiv 5 Jan 2021.

## Builds on (most cited first, 14 of 14)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Interpreting CNNs via Decision Trees](https://mlsanity.com/paper/1802.00121/) (2018)
- [Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions](https://mlsanity.com/paper/1901.03729/) (2019)
- [Beyond Sparsity: Tree Regularization of Deep Models for Interpretability](https://mlsanity.com/paper/1711.06178/) (2017)
- [Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations](https://mlsanity.com/paper/1702.07826/) (2017)
- [Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior](https://mlsanity.com/paper/1811.09722/) (2018)
- [Plan Explicability and Predictability for Robot Task Planning](https://mlsanity.com/paper/1511.08158/) (2015)
- [Leveraging Rationales to Improve Human Task Performance](https://mlsanity.com/paper/2002.04202/) (2020)
- [Endowing Robots with Longer-term Autonomy by Recovering from External Disturbances in Manipulation through Grounded Anomaly Classification and Recovery Policies](https://mlsanity.com/paper/1809.03979/) (2018)
- [Taking Recoveries to Task: Recovery-Driven Development for Recipe-based Robot Tasks](https://mlsanity.com/paper/2001.10386/) (2020)
- [Feature Guided Search for Creative Problem Solving Through Tool Construction](https://mlsanity.com/paper/2008.10685/) (2020)

## Built on by (most cited first, 30 of 32)

- [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023)
- [The Who in XAI: How AI Background Shapes Perceptions of AI Explanations](https://mlsanity.com/paper/2107.13509/) (2021)
- [A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings](https://mlsanity.com/paper/2405.13245/) (2024)
- [Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces](https://mlsanity.com/paper/2203.03254/) (2022)
- [XAIR: A Framework of Explainable AI in Augmented Reality](https://mlsanity.com/paper/2303.16292/) (2023)
- [Explainable Autonomous Robots: A Survey and Perspective](https://mlsanity.com/paper/2105.02658/) (2021)
- [Explainable Activity Recognition for Smart Home Systems](https://mlsanity.com/paper/2105.09787/) (2021)
- [Semantic-Based Explainable AI: Leveraging Semantic Scene Graphs and Pairwise Ranking to Explain Robot Failures](https://mlsanity.com/paper/2108.03554/) (2021)
- [Why did I fail? A Causal-based Method to Find Explanations for Robot Failures](https://mlsanity.com/paper/2204.04483/) (2022)
- [OG-SGG: Ontology-Guided Scene Graph Generation. A Case Study in Transfer Learning for Telepresence Robotics](https://mlsanity.com/paper/2202.10201/) (2022)
- [Security Considerations in AI-Robotics: A Survey of Current Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2310.08565/) (2023)
- [Surrogate Assisted Generation of Human-Robot Interaction Scenarios](https://mlsanity.com/paper/2304.13787/) (2023)
- [MIRROR: Differentiable Deep Social Projection for Assistive Human-Robot Communication](https://mlsanity.com/paper/2203.02877/) (2022)
- [Subgoal-Based Explanations for Unreliable Intelligent Decision Support Systems](https://mlsanity.com/paper/2201.04204/) (2022)
- [How Human-Centered Explainable AI Interface Are Designed and Evaluated: A Systematic Survey](https://mlsanity.com/paper/2403.14496/) (2024)
- [Towards Balancing Preference and Performance through Adaptive Personalized Explainability](https://mlsanity.com/paper/2504.13856/) (2025)
- [Towards Reconciling Usability and Usefulness of Explainable AI Methodologies](https://mlsanity.com/paper/2301.05347/) (2023)
- [Review on Fault Diagnosis and Fault-Tolerant Control Scheme for Robotic Manipulators: Recent Advances in AI, Machine Learning, and Digital Twin](https://mlsanity.com/paper/2402.02980/) (2024)
- [STAR: A Foundation Model-driven Framework for Robust Task Planning and Failure Recovery in Robotic Systems](https://mlsanity.com/paper/2503.06060/) (2025)
- [Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Tasks](https://mlsanity.com/paper/2312.05991/) (2023)
- [Multimodal Coherent Explanation Generation of Robot Failures](https://mlsanity.com/paper/2410.00659/) (2024)
- [Demonstration Based Explainable AI for Learning from Demonstration Methods](https://mlsanity.com/paper/2410.05715/) (2024)
- [KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis](https://mlsanity.com/paper/2604.07034/) (2026)
- [Imagining In-distribution States: How Predictable Robot Behavior Can Enable User Control Over Learned Policies](https://mlsanity.com/paper/2406.13711/) (2024)
- [BaTCAVe: Trustworthy Explanations for Robot Behaviors](https://mlsanity.com/paper/2409.10733/) (2024)
- [Actionable AI: Enabling Non Experts to Understand and Configure AI Systems](https://mlsanity.com/paper/2503.06803/) (2025)
- [Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Models](https://mlsanity.com/paper/2504.05477/) (2025)
- [Adapting Robot's Explanation for Failures Based on Observed Human Behavior in Human-Robot Collaboration](https://mlsanity.com/paper/2504.09717/) (2025)
- [Discovering Concept Directions from Diffusion-based Counterfactuals via Latent Clustering](https://mlsanity.com/paper/2505.07073/) (2025)
- [Effective Explanations for Belief-Desire-Intention Robots: When and What to Explain](https://mlsanity.com/paper/2507.02016/) (2025)

## Abstract (arXiv)

With the growing capabilities of intelligent systems, the integration of robots in our everyday life is increasing. However, when interacting in such complex human environments, the occasional failure of robotic systems is inevitable. The field of explainable AI has sought to make complex-decision making systems more interpretable but most existing techniques target domain experts. On the contrary, in many failure cases, robots will require recovery assistance from non-expert users. In this work, we introduce a new type of explanation, that explains the cause of an unexpected failure during an agent's plan execution to non-experts. In order for error explanations to be meaningful, we investigate what types of information within a set of hand-scripted explanations are most helpful to non-experts for failure and solution identification. Additionally, we investigate how such explanations can be autonomously generated, extending an existing encoder-decoder model, and generalized across environments. We investigate such questions in the context of a robot performing a pick-and-place manipulation task in the home environment. Our results show that explanations capturing the context of a failure and history of past actions, are the most effective for failure and solution identification among non-experts. Furthermore, through a second user evaluation, we verify that our model-generated explanations can generalize to an unseen office environment, and are just as effective as the hand-scripted explanations.