# Your Autoregressive Model Already Reveals the Causal Graph

arXiv: https://arxiv.org/abs/2602.01135 · HTML: https://mlsanity.com/paper/2602.01135/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 61 references that are in the map.
- First posted on arXiv 1 Feb 2026.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Detecting causal associations in large nonlinear time series datasets](https://mlsanity.com/paper/1702.07007/) (2017)
- [Transformer Hawkes Process](https://mlsanity.com/paper/2002.09291/) (2020)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data](https://mlsanity.com/paper/2006.10833/) (2020)
- [Can large language models build causal graphs?](https://mlsanity.com/paper/2303.05279/) (2023)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [Uncovering Causality from Multivariate Hawkes Integrated Cumulants](https://mlsanity.com/paper/1607.06333/) (2016)
- [CAUSE: Learning Granger Causality from Event Sequences using Attribution Methods](https://mlsanity.com/paper/2002.07906/) (2020)
- [A Survey on Causal Discovery Methods for I.I.D. and Time Series Data](https://mlsanity.com/paper/2303.15027/) (2023)
- [Causal Interpretation of Self-Attention in Pre-Trained Transformers](https://mlsanity.com/paper/2310.20307/) (2023)
- [Sparse and low-rank multivariate Hawkes processes](https://mlsanity.com/paper/1501.00725/) (2015)
- [Causal Hidden Markov Model for Time Series Disease Forecasting](https://mlsanity.com/paper/2103.16391/) (2021)
- [BSODA: A Bipartite Scalable Framework for Online Disease Diagnosis](https://mlsanity.com/paper/2012.01065/) (2020)
- [Structural Hawkes Processes for Learning Causal Structure from Discrete-Time Event Sequences](https://mlsanity.com/paper/2305.05986/) (2023)
- [Causal Discovery in Hawkes Processes by Minimum Description Length](https://mlsanity.com/paper/2206.06124/) (2022)
- [Estimating Causal Effects with the Neural Autoregressive Density Estimator](https://mlsanity.com/paper/2008.07283/) (2020)
- [Large-Scale Targeted Cause Discovery via Learning from Simulated Data](https://mlsanity.com/paper/2408.16218/) (2024)
- [Deep Autoregressive Models as Causal Inference Engines](https://mlsanity.com/paper/2409.18581/) (2024)
- [Harnessing Event Sensory Data for Error Pattern Prediction in Vehicles: A Language Model Approach](https://mlsanity.com/paper/2412.13041/) (2024)
- [Towards Practical Multi-label Causal Discovery in High-Dimensional Event Sequences via One-Shot Graph Aggregation](https://mlsanity.com/paper/2509.19112/) (2025)
- [One-Shot Multi-Label Causal Discovery in High-Dimensional Event Sequences](https://mlsanity.com/paper/2509.23213/) (2025)
- [Learning Causal Alignment for Reliable Disease Diagnosis](https://mlsanity.com/paper/2310.01766/) (2023)
- [Hawkes Models And Their Applications](https://mlsanity.com/paper/2405.10527/) (2024)

## Built on by (most cited first, 2 of 2)

- [EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systems](https://mlsanity.com/paper/2609.01360/) (2026)
- [Replicating TRACE: A Practitioner's Guide to Its Threshold and Particle Budget](https://mlsanity.com/paper/2609.01108/) (2026)

## Abstract (arXiv)

Autoregressive models trained via next-token prediction implicitly learn the conditional independence structure of their data-generating process. We exploit this observation to perform scalable causal discovery from a single observed sequence of discrete events -- without any task-specific retraining. Such single-stream settings arise naturally in vehicle diagnostics, manufacturing systems, and patient trajectories, yet they remain largely unsolved: the absence of repeated samples, massive event vocabularies, and long-range temporal dependencies render existing methods either inaccurate or computationally intractable. We introduce TRACE, a framework that repurposes any pretrained autoregressive model as a density estimator for conditional mutual information, the fundamental primitive for conditional independence testing. By constructing parallelized CI tests on GPUs, TRACE recovers both the sample-level time causal graph and its summary projection, scaling linearly with the vocabulary size while naturally handling delayed causal effects. Crucially, we prove that minimizing the standard cross-entropy pretraining loss directly minimizes an upper bound on the causal identification error, establishing a duality between sequence prediction and causal discovery. On nonlinear SCMs (|X| = 8000) and real-world vehicle diagnostic logs (|X| = 29100), TRACE is the first applicable method at this scale, outperforming the strongest baseline by over 20 F1 points.