mlsanityData to 14 Oct 2026

Model-Agnostic Policy Explanations with Large Language Models

arXiv 2504.05625 ↗First posted 8 Apr 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Intelligent agents, such as robots, are increasingly deployed in real-world, human-centric environments. To foster appropriate human trust and meet legal and ethical standards, these agents must be able to explain their behavior. However, state-of-the-art agents are typically driven by black-box models like deep neural networks, limiting their interpretability. We propose a method for generating natural language explanations of agent behavior based only on observed states and actions -- without access to the agent's underlying model. Our approach learns a locally interpretable surrogate model of the agent's behavior from observations, which then guides a large language model to generate plausible explanations with minimal hallucination. Empirical results show that our method produces explanations that are more comprehensible and correct than those from baselines, as judged by both language models and human evaluators. Furthermore, we find that participants in a user study more accurately predicted the agent's future actions when given our explanations, suggesting improved understanding of agent behavior.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

34 of 57 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2013Do Deep Nets Really Need to be Deep?cited by 474
  7. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  8. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  9. 2023GPTScore: Evaluate as You Desirecited by 361
  10. 2018Techniques for Interpretable Machine Learningcited by 238
  11. 2017Distilling a Neural Network Into a Soft Decision Treecited by 236
  12. 2018Programmatically Interpretable Reinforcement Learningcited by 193
22 more
  1. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  2. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 169
  3. 2020Explainable Reinforcement Learning: A Surveycited by 141
  4. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  5. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  6. 2019Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptionscited by 81
  7. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  8. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  9. 2020Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphscited by 46
  10. 2021SelfExplain: A Self-Explaining Architecture for Neural Text Classifierscited by 44
  11. 2017Interpretability via Model Extractioncited by 40
  12. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  13. 2023Concept Learning for Interpretable Multi-Agent Reinforcement Learningcited by 19
  14. 2020To what extent do human explanations of model behavior align with actual model behavior?cited by 18
  15. 2021Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learningcited by 16
  16. 2022Explainable Action Advising for Multi-Agent Reinforcement Learningcited by 10
  17. 2020BabyAI 1.1cited by 8
  18. 2021Interactive Explanations: Diagnosis and Repair of Reinforcement Learning Based Agent Behaviorscited by 8
  19. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
  20. 2016Natural Language Generation enhances human decision-making with uncertain informationcited by 5
  21. 2019Generating Justifications for Norm-Related Agent Decisionscited by 5
  22. 2025Reasoning-Grounded Natural Language Explanations for Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 59
  2. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Modelscited by 3
  3. 2026Data-Centric Interpretability for LLM-based Multi-Agent Reinforcement Learningcited by 1
  4. 2026Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oraclecited by 0