# Causal-INSIGHT: Probing Temporal Models to Extract Causal Structure

arXiv: https://arxiv.org/abs/2603.25473 · HTML: https://mlsanity.com/paper/2603.25473/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [Detecting causal associations in large nonlinear time series datasets](https://mlsanity.com/paper/1702.07007/) (2017)
- [DYNOTEARS: Structure Learning from Time-Series Data](https://mlsanity.com/paper/2002.00498/) (2020)
- [Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasets](https://mlsanity.com/paper/2003.03685/) (2020)
- [Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data](https://mlsanity.com/paper/2006.10833/) (2020)
- [Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causality](https://mlsanity.com/paper/1911.09879/) (2019)
- [CUTS: Neural Causal Discovery from Irregular Time-Series Data](https://mlsanity.com/paper/2302.07458/) (2023)
- [CausalFormer: An Interpretable Transformer for Temporal Causal Discovery](https://mlsanity.com/paper/2406.16708/) (2024)
- [Dynamic Causal Explanation Based Diffusion-Variational Graph Neural Network for Spatio-temporal Forecasting](https://mlsanity.com/paper/2305.09703/) (2023)
- [Causal discovery for time series from multiple datasets with latent contexts](https://mlsanity.com/paper/2306.12896/) (2023)

## Abstract (arXiv)

Understanding directed temporal interactions in multivariate time series is essential for interpreting complex dynamical systems and the predictive models trained on them. We present Causal-INSIGHT, a model-agnostic, post-hoc interpretation framework for extracting model-implied (predictor-dependent), directed, time-lagged influence structure from trained temporal predictors. Rather than inferring causal structure at the level of the data-generating process, Causal-INSIGHT analyzes how a fixed, pre-trained predictor responds to systematic, intervention-inspired input clamping applied at inference time. From these responses, we construct directed temporal influence signals that reflect the dependencies the predictor relies on for prediction, and introduce Qbic, a sparsity-aware graph selection criterion that balances predictive fidelity and structural complexity without requiring ground-truth graph labels. Experiments across synthetic, simulated, and realistic benchmarks show that Causal-INSIGHT generalizes across diverse backbone architectures, maintains competitive structural accuracy, and yields significant improvements in temporal delay localization when applied to existing predictors.