mlsanityData to 7 Oct 2026

Explainability Via Causal Self-Talk

arXiv 2211.09937 ↗First posted 17 Nov 2022cs.AI cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Explaining the behavior of AI systems is an important problem that, in practice, is generally avoided. While the XAI community has been developing an abundance of techniques, most incur a set of costs that the wider deep learning community has been unwilling to pay in most situations. We take a pragmatic view of the issue, and define a set of desiderata that capture both the ambitions of XAI and the practical constraints of deep learning. We describe an effective way to satisfy all the desiderata: train the AI system to build a causal model of itself. We develop an instance of this solution for Deep RL agents: Causal Self-Talk. CST operates by training the agent to communicate with itself across time. We implement this method in a simulated 3D environment, and show how it enables agents to generate faithful and semantically-meaningful explanations of their own behavior. Beyond explanations, we also demonstrate that these learned models provide new ways of building semantic control interfaces to AI systems.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 46 of its 86 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

46 of 86 references.

References · 46

  1. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  5. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  6. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  7. 2018Sanity Checks for Saliency Mapscited by 1,149
  8. 2020Concept Bottleneck Modelscited by 1,022
  9. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  10. 2022In-context Learning and Induction Headscited by 930
  11. 2019Attention is not Explanationcited by 916
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
34 more
  1. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  2. 2019Attention is not not Explanationcited by 629
  3. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  4. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  5. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  6. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  7. 2019Counterfactual Visual Explanationscited by 312
  8. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  9. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  10. 2016Learning to Navigate in Complex Environmentscited by 192
  11. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  12. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  13. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  14. 2020Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIcited by 166
  15. 2016Generating Visual Explanationscited by 156
  16. 2021PonderNet: Learning to Pondercited by 156
  17. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  18. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  19. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  20. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  21. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  22. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  23. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  24. 2015The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classificationcited by 76
  25. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  26. 2020Imitating Interactive Intelligencecited by 55
  27. 2018Generating Counterfactual Explanations with Natural Languagecited by 37
  28. 2019Generation of Policy-Level Explanations for Reinforcement Learningcited by 37
  29. 2020What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomescited by 20
  30. 2019Conditional Driving from Natural Language Instructionscited by 17
  31. 2020Using Unity to Help Solve Intelligencecited by 13
  32. 2019Terminal Prediction as an Auxiliary Task for Deep Reinforcement Learningcited by 11
  33. 2017Metacontrol for Adaptive Imagination-Based Optimizationcited by 8
  34. 2021Causal Analysis of Agent Behavior for AI Safetycited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5.

6 papers.

  1. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  2. 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learningcited by 18
  3. 2023Learning by Self-Explainingcited by 17
  4. 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learningcited by 11
  5. 2024Policy Learning with a Language Bottleneckcited by 4
  6. 2024Towards a Research Community in Interpretable Reinforcement Learning: the InterpPol Workshopcited by 0