# Interpretable Models for Granger Causality Using Self-explaining Neural Networks

arXiv: https://arxiv.org/abs/2101.07600 · HTML: https://mlsanity.com/paper/2101.07600/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 51 references that are in the map.
- First posted on arXiv 19 Jan 2021.

## Builds on (most cited first, 7 of 7)

- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data](https://mlsanity.com/paper/2006.10833/) (2020)
- [Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causality](https://mlsanity.com/paper/1911.09879/) (2019)
- [Neural Granger Causality](https://mlsanity.com/paper/1802.05842/) (2018)
- [Discovering Nonlinear Relations with Minimum Predictive Information Regularization](https://mlsanity.com/paper/2001.01885/) (2020)
- [Validity of time reversal for testing Granger causality](https://mlsanity.com/paper/1509.07636/) (2015)

## Built on by (most cited first, 30 of 45)

- [Concept Bottleneck Model with Additional Unsupervised Concepts](https://mlsanity.com/paper/2202.01459/) (2022)
- [Causal Recurrent Variational Autoencoder for Medical Time Series Generation](https://mlsanity.com/paper/2301.06574/) (2023)
- [Causal Discovery from Temporal Data: An Overview and New Perspectives](https://mlsanity.com/paper/2303.10112/) (2023)
- [Neural graphical modelling in continuous-time: consistency guarantees and algorithms](https://mlsanity.com/paper/2105.02522/) (2021)
- [A Survey on Causal Discovery Methods for I.I.D. and Time Series Data](https://mlsanity.com/paper/2303.15027/) (2023)
- [NTS-NOTEARS: Learning Nonparametric DBNs With Prior Knowledge](https://mlsanity.com/paper/2109.04286/) (2021)
- [Transferable Time-Series Forecasting under Causal Conditional Shift](https://mlsanity.com/paper/2111.03422/) (2021)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [Interpretation of Time-Series Deep Models: A Survey](https://mlsanity.com/paper/2305.14582/) (2023)
- [Deep Causal Learning: Representation, Discovery and Inference](https://mlsanity.com/paper/2211.03374/) (2022)
- [Deep Recurrent Modelling of Granger Causality with Latent Confounding](https://mlsanity.com/paper/2202.11286/) (2022)
- [Granger causal inference on DAGs identifies genomic loci regulating transcription](https://mlsanity.com/paper/2210.10168/) (2022)
- [Learning the Causal Structure of Networked Dynamical Systems under Latent Nodes and Structured Noise](https://mlsanity.com/paper/2312.05974/) (2023)
- [Learning interaction rules from multi-animal trajectories via augmented behavioral models](https://mlsanity.com/paper/2107.05326/) (2021)
- [CausalMMM: Learning Causal Structure for Marketing Mix Modeling](https://mlsanity.com/paper/2406.16728/) (2024)
- [Learning Flexible Time-windowed Granger Causality Integrating Heterogeneous Interventional Time Series Data](https://mlsanity.com/paper/2406.10419/) (2024)
- [Unifying Explainable Anomaly Detection and Root Cause Analysis in Dynamical Systems](https://mlsanity.com/paper/2502.12086/) (2025)
- [ClimateBench-M: A Multi-Modal Climate Data Benchmark with a Simple Generative Method](https://mlsanity.com/paper/2504.07394/) (2025)
- [Granger Causality using Neural Networks](https://mlsanity.com/paper/2208.03703/) (2022)
- [OCDB: Revisiting Causal Discovery with a Comprehensive Benchmark and Evaluation Framework](https://mlsanity.com/paper/2406.04598/) (2024)
- [UnCLe: Towards Scalable Dynamic Causal Discovery in Non-linear Temporal Systems](https://mlsanity.com/paper/2511.03168/) (2025)
- [Bayesian Spillover Graphs for Dynamic Networks](https://mlsanity.com/paper/2203.01912/) (2022)
- [Bayesian Vector AutoRegression with Factorised Granger-Causal Graphs](https://mlsanity.com/paper/2402.03614/) (2024)
- [Detecting Causality in the Frequency Domain with Cross-Mapping Coherence](https://mlsanity.com/paper/2407.20694/) (2024)
- [Exploring Neural Granger Causality with xLSTMs: Unveiling Temporal Dependencies in Complex Data](https://mlsanity.com/paper/2502.09981/) (2025)
- [Time Series Domain Adaptation via Latent Invariant Causal Mechanism](https://mlsanity.com/paper/2502.16637/) (2025)
- [An Interpretable Neural Network for Parameter Inference](https://mlsanity.com/paper/2106.05536/) (2021)
- [Jacobian Granger Causal Neural Networks for Analysis of Stationary and Nonstationary Data](https://mlsanity.com/paper/2205.09573/) (2022)
- [Nonlinear Permuted Granger Causality](https://mlsanity.com/paper/2308.06220/) (2023)
- [Algorithmic Recourse in Abnormal Multivariate Time Series](https://mlsanity.com/paper/2309.16896/) (2023)

## Abstract (arXiv)

Exploratory analysis of time series data can yield a better understanding of complex dynamical systems. Granger causality is a practical framework for analysing interactions in sequential data, applied in a wide range of domains. In this paper, we propose a novel framework for inferring multivariate Granger causality under nonlinear dynamics based on an extension of self-explaining neural networks. This framework is more interpretable than other neural-network-based techniques for inferring Granger causality, since in addition to relational inference, it also allows detecting signs of Granger-causal effects and inspecting their variability over time. In comprehensive experiments on simulated data, we show that our framework performs on par with several powerful baseline methods at inferring Granger causality and that it achieves better performance at inferring interaction signs. The results suggest that our framework is a viable and more interpretable alternative to sparse-input neural networks for inferring Granger causality.