# Explainable Goal-Driven Agents and Robots -- A Comprehensive Review

arXiv: https://arxiv.org/abs/2004.09705 · HTML: https://mlsanity.com/paper/2004.09705/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2020.

## Built on by (most cited first, 24 of 24)

- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [Because we have LLMs, we Can and Should Pursue Agentic Interpretability](https://mlsanity.com/paper/2506.12152/) (2025)
- [From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks](https://mlsanity.com/paper/2310.11884/) (2023)
- [GPT-in-the-Loop: Adaptive Decision-Making for Multiagent Systems](https://mlsanity.com/paper/2308.10435/) (2023)
- [A Closer Look at Reward Decomposition for High-Level Robotic Explanations](https://mlsanity.com/paper/2304.12958/) (2023)
- [PhenoAssistant: A Conversational Multi-Agent AI System for Automated Plant Phenotyping](https://mlsanity.com/paper/2504.19818/) (2025)
- [OntoLogX: Ontology-Guided Knowledge Graph Extraction from Cybersecurity Logs with Large Language Models](https://mlsanity.com/paper/2510.01409/) (2025)
- [XAI-N: Sensor-based Robot Navigation using Expert Policies and Decision Trees](https://mlsanity.com/paper/2104.10818/) (2021)
- [Advantage Actor-Critic with Reasoner: Explaining the Agent's Behavior from an Exploratory Perspective](https://mlsanity.com/paper/2309.04707/) (2023)
- [The Case for Negative Data: From Crash Reports to Counterfactuals for Reasonable Driving](https://mlsanity.com/paper/2509.18626/) (2025)
- [Explainable AI over the Internet of Things (IoT): Overview, State-of-the-Art and Future Directions](https://mlsanity.com/paper/2211.01036/) (2022)
- [Explainable AI is Responsible AI: How Explainability Creates Trustworthy and Socially Responsible Artificial Intelligence](https://mlsanity.com/paper/2312.01555/) (2023)
- [Enhancing Trust in Autonomous Agents: An Architecture for Accountability and Explainability through Blockchain and Large Language Models](https://mlsanity.com/paper/2403.09567/) (2024)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [Can AI detect pain and express pain empathy? A review from emotion recognition and a human-centered AI perspective](https://mlsanity.com/paper/2110.04249/) (2021)
- [Introspection-based Explainable Reinforcement Learning in Episodic and Non-episodic Scenarios](https://mlsanity.com/paper/2211.12930/) (2022)
- [Collision Probability Distribution Estimation via Temporal Difference Learning](https://mlsanity.com/paper/2407.20000/) (2024)
- [Effective Explanations for Belief-Desire-Intention Robots: When and What to Explain](https://mlsanity.com/paper/2507.02016/) (2025)
- [Enabling Transparent Cyber Threat Intelligence Combining Large Language Models and Domain Ontologies](https://mlsanity.com/paper/2509.00081/) (2025)
- [Multi-User Personalisation in Human-Robot Interaction: Resolving Preference Conflicts Using Gradual Argumentation](https://mlsanity.com/paper/2511.03576/) (2025)
- [A Survey on Quantitative Modeling of Trust in Online Social Networks](https://mlsanity.com/paper/2603.11054/) (2026)
- [Evaluating XAI Support From A Hierarchical Reinforcement Learning Policy in Human-Agent Collaboration](https://mlsanity.com/paper/2608.06381/) (2026)
- [Behaviora - A Conceptual Architecture for External and Internal Behavior of Robots and Agents](https://mlsanity.com/paper/2609.27536/) (2026)
- [Learning Adaptive Deliberation Control for Reliable Recognition with Multi-Agent Language Models](https://mlsanity.com/paper/2610.17840/) (2026)

## Abstract (arXiv)

Recent applications of autonomous agents and robots, such as self-driving cars, scenario-based trainers, exploration robots, and service robots have brought attention to crucial trust-related challenges associated with the current generation of artificial intelligence (AI) systems. AI systems based on the connectionist deep learning neural network approach lack capabilities of explaining their decisions and actions to others, despite their great successes. Without symbolic interpretation capabilities, they are black boxes, which renders their decisions or actions opaque, making it difficult to trust them in safety-critical applications. The recent stance on the explainability of AI systems has witnessed several approaches on eXplainable Artificial Intelligence (XAI); however, most of the studies have focused on data-driven XAI systems applied in computational sciences. Studies addressing the increasingly pervasive goal-driven agents and robots are still missing. This paper reviews approaches on explainable goal-driven intelligent agents and robots, focusing on techniques for explaining and communicating agents perceptual functions (example, senses, and vision) and cognitive reasoning (example, beliefs, desires, intention, plans, and goals) with humans in the loop. The review highlights key strategies that emphasize transparency, understandability, and continual learning for explainability. Finally, the paper presents requirements for explainability and suggests a roadmap for the possible realization of effective goal-driven explainable agents and robots.