# Interpretable Imitation Learning with Dynamic Causal Relations

arXiv: https://arxiv.org/abs/2310.00489 · HTML: https://mlsanity.com/paper/2310.00489/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 19 of its 51 references that are in the map.
- First posted on arXiv 30 Sep 2023.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 19 of 19)

- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [DAG-GNN: DAG Structure Learning with Graph Neural Networks](https://mlsanity.com/paper/1904.10098/) (2019)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data](https://mlsanity.com/paper/2006.10833/) (2020)
- [Towards Interpretable Reinforcement Learning Using Attention Augmented Agents](https://mlsanity.com/paper/1906.02500/) (2019)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [Graying the black box: Understanding DQNs](https://mlsanity.com/paper/1602.02658/) (2016)
- [Network Inference via the Time-Varying Graphical Lasso](https://mlsanity.com/paper/1703.01958/) (2017)
- [Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causality](https://mlsanity.com/paper/1911.09879/) (2019)
- [Mitigating Popularity Bias in Recommendation with Unbalanced Interactions: A Gradient Perspective](https://mlsanity.com/paper/2211.01154/) (2022)
- [Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning](https://mlsanity.com/paper/2307.03591/) (2023)
- [Latent Variable Time-varying Network Inference](https://mlsanity.com/paper/1802.03987/) (2018)
- [Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2306.07919/) (2023)
- [Improving Speech Related Facial Action Unit Recognition by Audiovisual Information Fusion](https://mlsanity.com/paper/1706.10197/) (2017)

## Abstract (arXiv)

Imitation learning, which learns agent policy by mimicking expert demonstration, has shown promising results in many applications such as medical treatment regimes and self-driving vehicles. However, it remains a difficult task to interpret control policies learned by the agent. Difficulties mainly come from two aspects: 1) agents in imitation learning are usually implemented as deep neural networks, which are black-box models and lack interpretability; 2) the latent causal mechanism behind agents' decisions may vary along the trajectory, rather than staying static throughout time steps. To increase transparency and offer better interpretability of the neural agent, we propose to expose its captured knowledge in the form of a directed acyclic causal graph, with nodes being action and state variables and edges denoting the causal relations behind predictions. Furthermore, we design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. Concretely, we conduct causal discovery from the perspective of Granger causality and propose a self-explainable imitation learning framework, method. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner. After the model is learned, we can obtain causal relations among states and action variables behind its decisions, exposing policies learned by it. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of the proposed method in learning the dynamic causal graphs for understanding the decision-making of imitation learning meanwhile maintaining high prediction accuracy.