mlsanityData to 7 Oct 2026

Discovering User-Interpretable Capabilities of Black-Box Planning Agents

arXiv 2107.13668 ↗First posted 28 Jul 2021cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Several approaches have been developed for answering users' specific questions about AI behavior and for assessing their core functionality in terms of primitive executable actions. However, the problem of summarizing an AI agent's broad capabilities for a user is comparatively new. This paper presents an algorithm for discovering from scratch the suite of high-level "capabilities" that an AI system with arbitrary internal planning algorithms/policies can perform. It computes conditions describing the applicability and effects of these capabilities in user-interpretable terms. Starting from a set of user-interpretable state properties, an AI agent, and a simulator that the agent can interact with, our algorithm returns a set of high-level capabilities with their parameterized descriptions. Empirical evaluation on several game-based scenarios shows that this approach efficiently learns descriptions of various types of AI agents in deterministic, fully observable settings. User studies show that such descriptions are easier to understand and reason with than the agent's primitive actions.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 26 of its 66 references that are in the map. First posted on arXiv 28 Jul 2021.

Builds on

26 of 66 references.

References · 26

  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  2. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  4. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  5. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  6. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  7. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 335
  8. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  9. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  10. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  11. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  12. 2017Visualizing and Understanding Atari Agentscited by 141
14 more
  1. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 77
  2. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  3. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  4. 2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
  5. 2017Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physicscited by 57
  6. 2018PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Makingcited by 51
  7. 2018General Video Game AI: a Multi-Track Framework for Evaluating Agents, Games and Content Generation Algorithmscited by 45
  8. 2019Learning First-Order Symbolic Representations for Planning from the Structure of the State Spacecited by 38
  9. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  10. 2018Composable Planning with Attributescited by 21
  11. 2020Learning Interpretable Concept-Based Models with Human Feedbackcited by 17
  12. 2017Efficient, Safe, and Probably Approximately Complete Learning of Action Modelscited by 16
  13. 2021Safe Learning of Lifted Action Modelscited by 16
  14. 2019Learning Portable Representations for High-Level Planningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 2; 2024: 0; 2025: 5; 2026: 1.

10 papers.

  1. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 55
  2. 2025Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selectioncited by 7
  3. 2025Language Models For Generalised PDDL Planning: Synthesising Sound and Programmatic Policiescited by 7
  4. 2023Autonomous Capability Assessment of Sequential Decision-Making Systems in Stochastic Settings (Extended Version)cited by 6
  5. 2025Neuro-Symbolic Imitation Learning: Discovering Symbolic Abstractions for Skill Learningcited by 4
  6. 2025Few-Shot Neuro-Symbolic Imitation Learning for Long-Horizon Planning and Actingcited by 2
  7. 2026Learning Lifted Action Models from Unsupervised Visual Tracescited by 2
  8. 2025HDDLGym: A Tool for Studying Multi-Agent Hierarchical Problems Defined in HDDL with OpenAI Gymcited by 1
  9. 2021JEDAI: A System for Skill-Aligned Explainable Robot Planningcited by 0
  10. 2023Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Makingcited by 0