# Transformers with Sparse Attention for Granger Causality

arXiv: https://arxiv.org/abs/2411.13264 · HTML: https://mlsanity.com/paper/2411.13264/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Causal Transformer for Estimating Counterfactual Outcomes](https://mlsanity.com/paper/2204.07258/) (2022)
- [How Transformers Learn Causal Structure with Gradient Descent](https://mlsanity.com/paper/2402.14735/) (2024)
- [CAUSE: Learning Granger Causality from Event Sequences using Attribution Methods](https://mlsanity.com/paper/2002.07906/) (2020)
- [A Survey on Causal Discovery Methods for I.I.D. and Time Series Data](https://mlsanity.com/paper/2303.15027/) (2023)
- [Causal Interpretation of Self-Attention in Pre-Trained Transformers](https://mlsanity.com/paper/2310.20307/) (2023)
- [Causal structure learning from time series: Large regression coefficients may predict causal links better in practice than small p-values](https://mlsanity.com/paper/2002.09573/) (2020)
- [Variable-lag Granger Causality for Time Series Analysis](https://mlsanity.com/paper/1912.10829/) (2019)
- [Granger Causality in Multi-variate Time Series using a Time Ordered Restricted Vector Autoregressive Model](https://mlsanity.com/paper/1511.03463/) (2015)
- [Learning Granger Causality from Instance-wise Self-attentive Hawkes Processes](https://mlsanity.com/paper/2402.03726/) (2024)
- [Sparse Transformer with Local and Seasonal Adaptation for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2312.06874/) (2023)

## Abstract (arXiv)

Temporal causal analysis means understanding the underlying causes behind observed variables over time. Deep learning based methods such as transformers are increasingly used to capture temporal dynamics and causal relationships beyond mere correlations. Recent works suggest self-attention weights of transformers as a useful indicator of causal links. We leverage this to propose a novel modification to the self-attention module to establish causal links between the variables of multivariate time-series data with varying lag dependencies. Our Sparse Attention Transformer captures causal relationships using a two-fold approach - performing temporal attention first followed by attention between the variables across the time steps masking them individually to compute Granger Causality indices. The key novelty in our approach is the ability of the model to assert importance and pick the most significant past time instances for its prediction task against manually feeding a fixed time lag value. We demonstrate the effectiveness of our approach via extensive experimentation on several synthetic benchmark datasets. Furthermore, we compare the performance of our model with the traditional Vector Autoregression based Granger Causality method that assumes fixed lag length.