mlsanityData to 7 Oct 2026

Explainable Reinforcement Learning Agents Using World Models

arXiv 2505.08073 ↗First posted 12 May 2025cs.AIcited by 1

On the map: World models → World models for language agents

Abstract (arXiv)

Explainable AI (XAI) systems have been proposed to help people understand how AI systems produce outputs and behaviors. Explainable Reinforcement Learning (XRL) has an added complexity due to the temporal nature of sequential decision-making. Further, non-AI experts do not necessarily have the ability to alter an agent or its policy. We introduce a technique for using World Models to generate explanations for Model-Based Deep RL agents. World Models predict how the world will change when actions are performed, allowing for the generation of counterfactual trajectories. However, identifying what a user wanted the agent to do is not enough to understand why the agent did something else. We augment Model-Based RL agents with a Reverse World Model, which predicts what the state of the world should have been for the agent to prefer a given counterfactual action. We show that explanations that show users what the world should have been like significantly increase their understanding of the agent policy. We hypothesize that our explanations can help users learn how to control the agents execution through by manipulating the environment.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 21 of its 34 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

21 of 34 references.

References · 21

  1. 2018World Modelscited by 1,444
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  6. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  7. 2019Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptionscited by 81
  8. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  9. 2020The Emerging Landscape of Explainable AI Planning and Decision Makingcited by 44
  10. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 38
  11. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  12. 2020Leveraging Rationales to Improve Human Task Performancecited by 20
9 more
  1. 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
  2. 2023State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understandingcited by 17
  3. 2021Inherently Explainable Reinforcement Learning in Natural Languagecited by 16
  4. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
  5. 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 12
  6. 2022Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agentscited by 10
  7. 2023RACCER: Towards Reachable and Certain Counterfactual Explanations for Reinforcement Learningcited by 6
  8. 2022Experiential Explanations for Reinforcement Learningcited by 5
  9. 2023Why Don't You Do Something About It? Outlining Connections between AI Explanations and User Actionscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 58

Similar, not cited

Close in wording, no citation either way

  1. 2022Decisions that Explain Themselves: A User-Centric Deep Reinforcement Learning Explanation System
  2. 2025A Survey on Explainable Deep Reinforcement Learning
  3. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models
  4. 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents
  5. 2023Explainable Reinforcement Learning via a Causal World Model
  6. 2026Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness
  7. 2024Mental Modeling of Reinforcement Learning Agents by Language Models
  8. 2023Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation