mlsanityData to 7 Oct 2026

Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learning

arXiv 2211.06376 ↗First posted 11 Nov 2022cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In recent years, advances in deep learning have resulted in a plethora of successes in the use of reinforcement learning (RL) to solve complex sequential decision tasks with high-dimensional inputs. However, existing systems lack the necessary mechanisms to provide humans with a holistic view of their competence, presenting an impediment to their adoption, particularly in critical applications where the decisions an agent makes can have significant consequences. Yet, existing RL-based systems are essentially competency-unaware in that they lack the necessary interpretation mechanisms to allow human operators to have an insightful, holistic view of their competency. In this paper, we extend a recently-proposed framework for explainable RL that is based on analyses of "interestingness." Our new framework provides various measures of RL agent competence stemming from interestingness analysis and is applicable to a wide range of RL algorithms. We also propose novel mechanisms for assessing RL agents' competencies that: 1) identify agent behavior patterns and competency-controlling conditions by clustering agent behavior traces solely using interestingness data; and 2) identify the task elements mostly responsible for an agent's behavior, as measured through interestingness, by performing global and local analyses using SHAP values. Overall, our tools provide insights about RL agent competence, both their capabilities and limitations, enabling users to make more informed decisions about interventions, additional training, and other interactions in collaborative human-machine settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 44 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

23 of 44 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  9. 2018Metrics for Explainable AI: Challenges and Prospectscited by 257
  10. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  11. 2017Visualizing and Understanding Atari Agentscited by 141
  12. 2020Explainable Reinforcement Learning: A Surveycited by 139
11 more
  1. 2020Explainability in Deep Reinforcement Learningcited by 96
  2. 2016Graying the black box: Understanding DQNscited by 68
  3. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  4. 2018Establishing Appropriate Trust via Critical Statescited by 47
  5. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  6. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 38
  7. 2018Learning Finite State Representations of Recurrent Policy Networkscited by 37
  8. 2017Enabling Robots to Communicate their Objectivescited by 34
  9. 2019Exploring Computational User Models for Agent Policy Summarizationcited by 29
  10. 2022Outcome-Guided Counterfactuals for Reinforcement Learning Agents from a Jointly Trained Generative Latent Spacecited by 2
  11. 2022A Framework for Understanding and Visualizing Strategies of RL Agentscited by 2

Built on by

Not cited yet.