mlsanityData to 14 Oct 2026

CaTs and DAGs: Integrating Directed Acyclic Graphs with Transformers for Causally Constrained Predictions

arXiv 2410.14485 ↗First posted 18 Oct 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Artificial Neural Networks (ANNs), including fully-connected networks and transformers, are highly flexible and powerful function approximators, widely applied in fields like computer vision and natural language processing. However, their inability to inherently respect causal structures can limit their robustness, making them vulnerable to covariate shift and difficult to interpret/explain. This poses significant challenges for their reliability in real-world applications. In this paper, we introduce Causal Transformers (CaTs), a general model class designed to operate under predefined causal constraints, as specified by a Directed Acyclic Graph (DAG). CaTs retain the powerful function approximation abilities of traditional neural networks while adhering to the underlying structural constraints, improving robustness, reliability, and interpretability at inference time. This approach opens new avenues for deploying neural networks in more demanding, real-world scenarios where robustness and explainability is critical.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 55 of its 120 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

55 of 120 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Layer Normalizationcited by 5,669
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2019Invariant Risk Minimizationcited by 1,854
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
38 more
  1. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
  2. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 600
  3. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 410
  4. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 307
  5. 2019Causal Confusion in Imitation Learningcited by 288
  6. 2019Adapting Neural Networks for the Estimation of Treatment Effectscited by 287
  7. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  8. 2015Recursive Partitioning for Heterogeneous Causal Effectscited by 255
  9. 2019Causality for Machine Learningcited by 243
  10. 2015The Variational Fair Autoencodercited by 223
  11. 2021D'ya like DAGs? A Survey on Structure Learning and Causal Discoverycited by 211
  12. 2020A Survey on Causal Inferencecited by 202
  13. 2019Causality matters in medical imagingcited by 193
  14. 2015Deep Kalman Filterscited by 166
  15. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  16. 2017CausalGAN: Learning Causal Implicit Generative Models with Adversarial Trainingcited by 152
  17. 2019On the Fairness of Disentangled Representationscited by 144
  18. 2022Causal Transformer for Estimating Counterfactual Outcomescited by 123
  19. 2018Disentangled Sequential Autoencodercited by 120
  20. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 120
  21. 2024Evaluating the World Model Implicit in a Generative Modelcited by 113
  22. 2021The Causal-Neural Connection: Expressiveness, Learnability, and Inferencecited by 98
  23. 2021Directed Acyclic Graph Network for Conversational Emotion Recognitioncited by 97
  24. 2017Causal Structure Learningcited by 84
  25. 2018Hyperparameters and Tuning Strategies for Random Forestcited by 84
  26. 2017Double/Debiased/Neyman Machine Learning of Treatment Effectscited by 76
  27. 2013A Deep and Tractable Density Estimatorcited by 68
  28. 2020Treatment effect estimation with disentangled latent factorscited by 63
  29. 2019Learning Individual Causal Effects from Networked Observational Datacited by 56
  30. 2017Nested Markov Properties for Acyclic Directed Mixed Graphscited by 39
  31. 2020Graphical Normalizing Flowscited by 37
  32. 2022Transformers over Directed Acyclic Graphscited by 23
  33. 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20
  34. 2019A Potential Outcomes Calculus for Identifying Conditional Path-Specific Effectscited by 19
  35. 2019DynaNet: Neural Kalman Dynamical Model for Motion Estimation and Predictioncited by 13
  36. 2023Structured Neural Networks for Density Estimation and Causal Inferencecited by 13
  37. 2022Neural Topological Ordering for Computation Graphscited by 12
  38. 2024Smoke and Mirrors in Causal Downstream Taskscited by 12

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.