mlsanityData to 14 Oct 2026

Causal Transformer for Estimating Counterfactual Outcomes

arXiv 2204.07258 ↗First posted 14 Apr 2022cs.LG stat.MLcited by 123

Not on the map. Why?

Abstract (arXiv)

Estimating counterfactual outcomes over time from observational data is relevant for many applications (e.g., personalized medicine). Yet, state-of-the-art methods build upon simple long short-term memory (LSTM) networks, thus rendering inferences for complex, long-range dependencies challenging. In this paper, we develop a novel Causal Transformer for estimating counterfactual outcomes over time. Our model is specifically designed to capture complex, long-range dependencies among time-varying confounders. For this, we combine three transformer subnetworks with separate inputs for time-varying covariates, previous treatments, and previous outcomes into a joint network with in-between cross-attentions. We further develop a custom, end-to-end training procedure for our Causal Transformer. Specifically, we propose a novel counterfactual domain confusion loss to address confounding bias: it aims to learn adversarial balanced representations, so that they are predictive of the next outcome but non-predictive of the current treatment assignment. We evaluate our Causal Transformer based on synthetic and real-world datasets, where it achieves superior performance over current baselines. To the best of our knowledge, this is the first work proposing transformer-based architecture for estimating counterfactual outcomes from longitudinal data.

Key facts. Cited by 123 papers in the map (data through 14 Oct 2026). Builds on 33 of its 68 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

33 of 68 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2016Layer Normalizationcited by 5,669
  8. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  9. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  10. 2017Adversarial Discriminative Domain Adaptationcited by 1,311
  11. 2018Self-Attention with Relative Position Representationscited by 1,120
  12. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 600
21 more
  1. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  2. 2016Learning Representations for Counterfactual Inferencecited by 329
  3. 2015Simultaneous Deep Transfer Across Domains and Taskscited by 206
  4. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  5. 2020Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representationscited by 155
  6. 2021Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithmscited by 144
  7. 2019MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-IIIcited by 122
  8. 2016Variational Fourier features for Gaussian processescited by 85
  9. 2018The Unusual Effectiveness of Averaging in GAN Trainingcited by 73
  10. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  11. 2019Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solvingcited by 35
  12. 2022Estimating average causal effects from patient trajectoriescited by 29
  13. 2021Sequential Deconfounding for Causal Inference with Unobserved Confounderscited by 24
  14. 2018Input Combination Strategies for Multi-Source Transformer Decodercited by 22
  15. 2019DropAttention: A Regularization Method for Fully-Connected Self-Attention Networkscited by 20
  16. 2017Bayesian Nonparametric Causal Inference: Information Rates and Learning Algorithmscited by 16
  17. 2021Deconfounding Temporal Autoencoder: Estimating Treatment Effects over Time Using Noisy Proxiescited by 16
  18. 2020G-Net: A Deep Learning Approach to G-computation for Counterfactual Outcome Prediction Under Dynamic Treatment Regimescited by 13
  19. 2021Disentangled Counterfactual Recurrent Networks for Treatment Effect Inference over Timecited by 9
  20. 2022Estimating Conditional Average Treatment Effects with Missing Treatment Informationcited by 7
  21. 2018Learning Optimal Policies from Observational Datacited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 24; 2024: 30; 2025: 25; 2026: 35.

123 papers.

Sort:

By area:

  1. 2024Causal machine learning for predicting treatment outcomescited by 74
  2. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  3. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 70
  4. 2025Foundation Models for Causal Inference via Prior-Data Fitted Networkscited by 32
  5. 2023Bayesian Neural Controlled Differential Equations for Treatment Effect Estimationcited by 26
  6. 2023Sharp Bounds for Generalized Causal Sensitivity Analysiscited by 24
  7. 2023Causal Deep Learningcited by 20
  8. 2023Bounds on Representation-Induced Confounding Bias for Treatment Effect Estimationcited by 20
  9. 2024Model-agnostic meta-learners for estimating heterogeneous treatment effects over timecited by 19
  10. 2022Estimating individual treatment effects under unobserved confounding using binary instrumentscited by 18
  11. 2023Counterfactual Generative Models for Time-Varying Treatmentscited by 17
  12. 2023Towards Causal Foundation Model: on Duality between Causal Inference and Attentioncited by 17
38 more
  1. 2023CF-GODE: Continuous-Time Causal Inference for Multi-Agent Dynamical Systemscited by 16
  2. 2024Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patchingcited by 16
  3. 2023Reliable Off-Policy Learning for Dosage Combinationscited by 15
  4. 2023Accounting For Informative Sampling When Learning to Forecast Treatment Outcomes Over Timecited by 15
  5. 2024Automatically Learning Hybrid Digital Twins of Dynamical Systemscited by 14
  6. 2023Deep Learning based Forecasting: a case study from the online fashion industrycited by 12
  7. 2024ODE Discovery for Longitudinal Heterogeneous Treatment Effects Inferencecited by 12
  8. 2022Deep Causal Learning: Representation, Discovery and Inferencecited by 11
  9. 2023Fair Off-Policy Learning from Observational Datacited by 11
  10. 2023Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learningcited by 11
  11. 2022A Survey of Deep Causal Models and Their Industrial Applicationscited by 10
  12. 2023Instrumental Variable Estimation for Causal Inference in Longitudinal Data with Time-Dependent Latent Confounderscited by 10
  13. 2024Longitudinal Targeted Minimum Loss-based Estimation with Temporal-Difference Heterogeneous Transformercited by 10
  14. 2024IGC-Net for conditional average potential outcome estimation over timecited by 10
  15. 2023Clinical Decision Transformer: Intended Treatment Recommendation through Goal Promptingcited by 8
  16. 2024Causal Graph ODE: Continuous Treatment Effect Modeling in Multi-agent Dynamical Systemscited by 8
  17. 2024G-Transformer: Counterfactual Outcome Prediction under Dynamic and Time-varying Treatment Regimescited by 8
  18. 2025Treatment Effect Estimation for Optimal Decision-Makingcited by 8
  19. 2025G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibrationcited by 8
  20. 2024Causal Contrastive Learning for Counterfactual Regression Over Timecited by 7
  21. 2024Stabilized Neural Prediction of Potential Outcomes in Continuous Timecited by 7
  22. 2024Counterfactual Causal Inference in Natural Language with Large Language Modelscited by 7
  23. 2026Interventional Time Series Priors for Causal Foundation Modelscited by 7
  24. 2023A Survey of Methods, Challenges and Perspectives in Causalitycited by 6
  25. 2023Learning the Causal Structure of Networked Dynamical Systems under Latent Nodes and Structured Noisecited by 6
  26. 2024Rough Transformers for Continuous and Efficient Time-Series Modellingcited by 6
  27. 2024Constructing Confidence Intervals for Average Treatment Effects from Multiple Datasetscited by 6
  28. 2022Time Series Causal Link Estimation under Hidden Confounding using Knockoff Interventionscited by 5
  29. 2024Multi-task Heterogeneous Graph Learning on Electronic Health Recordscited by 5
  30. 2025Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessingcited by 5
  31. 2025Orthogonal Representation Learning for Estimating Causal Quantitiescited by 5
  32. 2025GST-UNet: A Neural Framework for Spatiotemporal Causal Inference with Time-Varying Confoundingcited by 5
  33. 2022Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya-Watson Regressioncited by 4
  34. 20233D-CLFusion: Fast Text-to-3D Rendering with Contrastive Latent Diffusioncited by 4
  35. 2025Adjustment for Confounding using Pre-Trained Representationscited by 4
  36. 2025Transformer-Based Spatial-Temporal Counterfactual Outcomes Estimationcited by 4
  37. 2025Counterfactual Probabilistic Diffusion with Expert Modelscited by 4
  38. 2026Medical world models in healthcare: foundations, applications, and challenges for trustworthy clinical translationcited by 4

50 most cited of 123 All 123 (JSON)