mlsanityData to 14 Oct 2026

Explaining Agent Behavior with Large Language Models

arXiv 2309.10346 ↗First posted 19 Sep 2023cs.LG cs.AI cs.CLcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Intelligent agents such as robots are increasingly deployed in real-world, safety-critical settings. It is vital that these agents are able to explain the reasoning behind their decisions to human counterparts, however, their behavior is often produced by uninterpretable models such as deep neural networks. We propose an approach to generate natural language explanations for an agent's behavior based only on observations of states and actions, agnostic to the underlying model representation. We show how a compact representation of the agent's behavior can be learned and used to produce plausible explanations with minimal hallucination while affording user interaction with a pre-trained large language model. Through user studies and empirical experiments, we show that our approach generates explanations as helpful as those generated by a human domain expert while enabling beneficial interactions such as clarification and counterfactual queries.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 36 of its 52 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

36 of 52 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  5. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  6. 2013Do Deep Nets Really Need to be Deep?cited by 474
  7. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  8. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  9. 2018Techniques for Interpretable Machine Learningcited by 238
  10. 2017Distilling a Neural Network Into a Soft Decision Treecited by 236
  11. 2018Programmatically Interpretable Reinforcement Learningcited by 193
  12. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
24 more
  1. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 169
  2. 2020Explainable Reinforcement Learning: A Surveycited by 141
  3. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  4. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  5. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  6. 2019Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptionscited by 81
  7. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  8. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  9. 2020Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphscited by 46
  10. 2017Interpretability via Model Extractioncited by 40
  11. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  12. 2022A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policiescited by 26
  13. 2021Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanationscited by 24
  14. 2023Concept Learning for Interpretable Multi-Agent Reinforcement Learningcited by 19
  15. 2020To what extent do human explanations of model behavior align with actual model behavior?cited by 18
  16. 2021Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learningcited by 16
  17. 2022Explainable Action Advising for Multi-Agent Reinforcement Learningcited by 10
  18. 2021Interactive Explanations: Diagnosis and Repair of Reinforcement Learning Based Agent Behaviorscited by 8
  19. 2023Sample-Efficient Learning of Novel Visual Conceptscited by 8
  20. 2016Natural Language Generation enhances human decision-making with uncertain informationcited by 5
  21. 2019Generating Justifications for Norm-Related Agent Decisionscited by 5
  22. 2023Demonstrating Large-Scale Package Manipulation via Learned Metrics of Pick Successcited by 4
  23. 2020Learning Whole-Body Human-Robot Haptic Interaction in Social Contextscited by 3
  24. 2023Introspective Action Advising for Interpretable Transfer Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanationscited by 17
  2. 2025Mind the XAI Gap: A Human-Centered LLM Framework for Democratizing Explainable AIcited by 0
  3. 2026Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oraclecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation
  2. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations
  3. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
  4. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models
  5. 2026MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations
  6. 2025Why do AI agents communicate in human language?
  7. 2024Explaining Autonomy: Enhancing Human-Robot Interaction through Explanation Generation with Large Language Models
  8. 2022From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent