mlsanityData to 14 Oct 2026

Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes

arXiv 2312.11118 ↗First posted 18 Dec 2023cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Explainable reinforcement learning (XRL) methods aim to help elucidate agent policies and decision-making processes. The majority of XRL approaches focus on local explanations, seeking to shed light on the reasons an agent acts the way it does at a specific world state. While such explanations are both useful and necessary, they typically do not portray the outcomes of the agent's selected choice of action. In this work, we propose ``COViz'', a new local explanation method that visually compares the outcome of an agent's chosen action to a counterfactual one. In contrast to most local explanations that provide state-limited observations of the agent's motivation, our method depicts alternative trajectories the agent could have taken from the given state and their outcomes. We evaluated the usefulness of COViz in supporting people's understanding of agents' preferences and compare it with reward decomposition, a local explanation method that describes an agent's expected utility for different actions by decomposing it into meaningful reward types. Furthermore, we examine the complementary benefits of integrating both methods. Our results show that such integration significantly improved participants' performance.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

18 of 37 references.

References · 18

  1. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  2. 2018Metrics for Explainable AI: Challenges and Prospectscited by 260
  3. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  4. 2017Visualizing and Understanding Atari Agentscited by 141
  5. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 139
  6. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  7. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 58
  8. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  9. 2019Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attributioncited by 50
  10. 2018Establishing Appropriate Trust via Critical Statescited by 48
  11. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  12. 2017Enabling Robots to Communicate their Objectivescited by 35
6 more
  1. 2022A Survey of Explainable Reinforcement Learningcited by 31
  2. 2020Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Mapscited by 30
  3. 2019Exploring Computational User Models for Agent Policy Summarizationcited by 29
  4. 2019Explaining Reinforcement Learning to Mere Mortals: An Empirical Studycited by 21
  5. 2020Contrastive Explanations for Reinforcement Learning via Embedded Self Predictionscited by 13
  6. 2021"I Don't Think So": Summarizing Policy Disagreements for Agent Comparisoncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 6.

14 papers.

Sort:

By area:

  1. 2024Enhancing RL Safety with Counterfactual LLM Reasoningcited by 6
  2. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Modelscited by 3
  3. 2025Interactive Explanations for Reinforcement-Learning Agentscited by 2
  4. 2025Explainable Reinforcement Learning Agents Using World Modelscited by 1
  5. 2025Counterfactual Explanations for Continuous Action Reinforcement Learningcited by 1
  6. 2026Joint MDPs and Reinforcement Learning in Coupled-Dynamics Environmentscited by 1
  7. 2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0
  8. 2025Ghost Policies: A New Paradigm for Understanding and Learning from Failure in Deep Reinforcement Learningcited by 0
  9. 2025STACHE: Local Black-Box Explanations for Reinforcement Learning Policiescited by 0
  10. 2026Contrastive explanations of BDI agentscited by 0
  11. 2026Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillationcited by 0
  12. 2026Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learningcited by 0
2 more
  1. 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agentscited by 0
  2. 2026Visualizing the World Model in an Agent's Value Functioncited by 0