mlsanityData to 7 Oct 2026

Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations

arXiv 1912.09007 ↗First posted 19 Dec 2019cs.LG cs.AI cs.HCcited by 50

Not on the map. Why?

Abstract (arXiv)

We propose an explainable reinforcement learning (XRL) framework that analyzes an agent's history of interaction with the environment to extract interestingness elements that help explain its behavior. The framework relies on data readily available from standard RL algorithms, augmented with data that can easily be collected by the agent while learning. We describe how to create visual summaries of an agent's behavior in the form of short video-clips highlighting key interaction moments, based on the proposed elements. We also report on a user study where we evaluated the ability of humans to correctly perceive the aptitude of agents with different characteristics, including their capabilities and limitations, given visual summaries automatically generated by our framework. The results show that the diversity of aspects captured by the different interestingness elements is crucial to help humans correctly understand an agent's strengths and limitations in performing a task, and determine when it might need adjustments to improve its performance.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 18; 2023: 9; 2024: 2; 2025: 5; 2026: 5.

50 papers.

Sort:

By area:

  1. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  2. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  3. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 38
  4. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  5. 2021Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Surveycited by 24
  6. 2021A Survey on AI Assurancecited by 16
  7. 2021Explainable Autonomous Robots: A Survey and Perspectivecited by 15
  8. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
  9. 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 12
  10. 2022Explainability in reinforcement learning: perspective and positioncited by 11
  11. 2021Learning Symbolic Rules for Interpretable Deep Reinforcement Learningcited by 9
  12. 2022Sibyl: Adaptive and Extensible Data Placement in Hybrid Storage Systems Using Online Reinforcement Learningcited by 8
38 more
  1. 2021"I Don't Think So": Summarizing Policy Disagreements for Agent Comparisoncited by 6
  2. 2021Contrastive Explanations for Comparing Preferences of Reinforcement Learning Agentscited by 5
  3. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 5
  4. 2022Explaining Online Reinforcement Learning Decisions of Self-Adaptive Systemscited by 5
  5. 2022ReCCoVER: Detecting Causal Confusion for Explainable Reinforcement Learningcited by 4
  6. 2022There is no Accuracy-Interpretability Tradeoff in Reinforcement Learning for Mazescited by 4
  7. 2022ProtoX: Explaining a Reinforcement Learning Agent via Prototypingcited by 4
  8. 2024ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policiescited by 4
  9. 2022Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysiscited by 3
  10. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  11. 2023A User Study on Explainable Online Reinforcement Learning for Adaptive Systemscited by 3
  12. 2021AutoPreview: A Framework for Autopilot Behavior Understandingcited by 2
  13. 2022Outcome-Guided Counterfactuals for Reinforcement Learning Agents from a Jointly Trained Generative Latent Spacecited by 2
  14. 2022A Framework for Understanding and Visualizing Strategies of RL Agentscited by 2
  15. 2022Explainable Reinforcement Learning on Financial Stock Trading using SHAPcited by 2
  16. 2023Iterative Reward Shaping using Human Feedback for Correcting Reward Misspecificationcited by 2
  17. 2025Interactive Explanations for Reinforcement-Learning Agentscited by 2
  18. 2025TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Modelscited by 2
  19. 2026On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVRcited by 2
  20. 2022Automated Gadget Discovery in Sciencecited by 1
  21. 2023An AI Chatbot for Explaining Deep Reinforcement Learning Decisions of Service-oriented Systemscited by 1
  22. 2023Utilizing Explainability Techniques for Reinforcement Learning Model Assurancecited by 1
  23. 2024Structure and Reduction of MCTS for Explainable-AIcited by 1
  24. 2025Surrogate Fitness Metrics for Interpretable Reinforcement Learningcited by 1
  25. 2021The Atari Data Scrapercited by 0
  26. 2022System-Agnostic Meta-Learning for MDP-based Dynamic Scheduling via Descriptive Policycited by 0
  27. 2022Incorporating Rivalry in Reinforcement Learning for a Competitive Gamecited by 0
  28. 2022Symbolic Distillation for Learned TCP Congestion Controlcited by 0
  29. 2022Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learningcited by 0
  30. 2023ASQ-IT: Interactive Explanations for Reinforcement-Learning Agentscited by 0
  31. 2023Conveying Autonomous Robot Capabilities through Contrasting Behaviour Summariescited by 0
  32. 2023IxDRL: A Novel Explainable Deep Reinforcement Learning Toolkit based on Analyses of Interestingnesscited by 0
  33. 2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0
  34. 2025Domain-driven Metrics for Reinforcement Learning: A Case Study on Epidemic Control using Agent-based Simulationcited by 0
  35. 2026Neetyabhas: A Framework for Uncertainty-Aware Public Policy Optimization in Rational Agent-Based Modelscited by 0
  36. 2026Self-Explainability in Self-Adaptive and Self-Organising Systems: Status and Research Directionscited by 0
  37. 2026Explaining Reinforcement Learning Agents via Inductive Logic Programmingcited by 0
  38. 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agentscited by 0