mlsanityData to 14 Oct 2026

TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models

arXiv 2509.04809 ↗First posted 5 Sep 2025cs.AI cs.HCcited by 3

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Explainable Reinforcement Learning (XRL) has emerged as a promising approach in improving the transparency of Reinforcement Learning (RL) agents. However, there remains a gap between complex RL policies and domain experts, due to the limited comprehensibility of XRL results and isolated coverage of current XRL approaches that leave users uncertain about which tools to employ. To address these challenges, we introduce TalkToAgent, a multi-agent Large Language Models (LLM) framework that delivers interactive, natural language explanations for RL policies. The architecture with five specialized LLM agents (Coordinator, Explainer, Coder, Evaluator, and Debugger) enables TalkToAgent to automatically map user queries to relevant XRL tools and clarify an agent's actions in terms of either key state variables, expected outcomes, or counterfactual explanations. Moreover, our approach extends previous counterfactual explanations by deriving alternative scenarios from qualitative behavioral descriptions, or even new rule-based policies. We validated TalkToAgent on quadruple-tank process control problem, a well-known nonlinear control benchmark. Results demonstrated that TalkToAgent successfully mapped user queries into XRL tasks with high accuracy, and coder-debugger interactions minimized failures in counterfactual generation. Furthermore, qualitative evaluation confirmed that TalkToAgent effectively interpreted agent's actions and contextualized their meaning within the problem domain.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 33 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

21 of 33 references.

References · 21

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  4. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  5. 2020Explainable Reinforcement Learning: A Surveycited by 141
  6. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  7. 2016Graying the black box: Understanding DQNscited by 68
  8. 2022TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversationscited by 66
  9. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  10. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  11. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  12. 2018Establishing Appropriate Trust via Critical Statescited by 48
9 more
  1. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  2. 2020What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomescited by 20
  3. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 14
  4. 2020Contrastive Explanations for Reinforcement Learning via Embedded Self Predictionscited by 13
  5. 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 13
  6. 2024PC-Gym: Benchmark Environments For Process Control Problemscited by 10
  7. 2025Explainable Reinforcement Learning via Temporal Policy Decompositioncited by 5
  8. 2025Model-Agnostic Policy Explanations with Large Language Modelscited by 4
  9. 2024Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimatorscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026A MARL Centered Reference Architecture for Large Language Model Augmentation in Smart Manufacturingcited by 0
  2. 2026Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oraclecited by 0
  3. 2026MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning
  2. 2025Explainable Reinforcement Learning Agents Using World Models
  3. 2022Decisions that Explain Themselves: A User-Centric Deep Reinforcement Learning Explanation System
  4. 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents
  5. 2025Interactive Explanations for Reinforcement-Learning Agents
  6. 2023Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation
  7. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  8. 2025A Survey on Explainable Deep Reinforcement Learning