# Do-PFN: In-Context Learning for Causal Effect Estimation

arXiv: https://arxiv.org/abs/2506.06039 · HTML: https://mlsanity.com/paper/2506.06039/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 70 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [FLAML: A Fast and Lightweight AutoML Library](https://mlsanity.com/paper/1911.04706/) (2019)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [Flow Matching for Scalable Simulation-Based Inference](https://mlsanity.com/paper/2305.17161/) (2023)
- [DoWhy: An End-to-End Library for Causal Inference](https://mlsanity.com/paper/2011.04216/) (2020)
- [Amortized Inference for Causal Structure Learning](https://mlsanity.com/paper/2205.12934/) (2022)
- [All-in-one simulation-based inference](https://mlsanity.com/paper/2404.09636/) (2024)
- [PFNs4BO: In-Context Learning for Bayesian Optimization](https://mlsanity.com/paper/2305.17535/) (2023)
- [Learning to Induce Causal Structure](https://mlsanity.com/paper/2204.04875/) (2022)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)
- [RealCause: Realistic Causal Inference Benchmarking](https://mlsanity.com/paper/2011.15007/) (2020)
- [Causal de Finetti: On the Identification of Invariant Causal Structure in Exchangeable Data](https://mlsanity.com/paper/2203.15756/) (2022)
- [DoWhy-GCM: An extension of DoWhy for causal inference in graphical causal models](https://mlsanity.com/paper/2206.06821/) (2022)
- [Mixture of In-Context Prompters for Tabular PFNs](https://mlsanity.com/paper/2405.16156/) (2024)
- [Can Transformers Learn Full Bayesian Inference in Context?](https://mlsanity.com/paper/2501.16825/) (2025)
- [In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization](https://mlsanity.com/paper/2404.16795/) (2024)
- [A Meta-Learning Approach to Bayesian Causal Discovery](https://mlsanity.com/paper/2412.16577/) (2024)

## Built on by (most cited first, 30 of 51)

- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [Foundation Models for Causal Inference via Prior-Data Fitted Networks](https://mlsanity.com/paper/2506.10914/) (2025)
- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [Relational In-Context Learning via Synthetic Pre-training with Structural Prior](https://mlsanity.com/paper/2603.03805/) (2026)
- [From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detection](https://mlsanity.com/paper/2602.03018/) (2026)
- [Use What You Know: Causal Foundation Models with Partial Graphs](https://mlsanity.com/paper/2602.14972/) (2026)
- [Interventional Time Series Priors for Causal Foundation Models](https://mlsanity.com/paper/2603.11090/) (2026)
- [Survival In-Context: Amortized Bayesian Survival Analysis via Prior-Fitted Networks](https://mlsanity.com/paper/2603.29475/) (2026)
- [Does TabPFN Understand Causal Structures?](https://mlsanity.com/paper/2511.07236/) (2025)
- [Causal Foundation Models with Continuous Treatments](https://mlsanity.com/paper/2605.15133/) (2026)
- [Towards Continuous-time Causal Foundation Models](https://mlsanity.com/paper/2605.28880/) (2026)
- [MapPFN: Learning Causal Perturbation Maps in Context](https://mlsanity.com/paper/2601.21092/) (2026)
- [Improving TabPFN's Synthetic Data Generation by Integrating Causal Structure](https://mlsanity.com/paper/2603.10254/) (2026)
- [Amortizing Causal Sensitivity Analysis via Prior Data-Fitted Networks](https://mlsanity.com/paper/2605.10590/) (2026)
- [IV-ICL: Bounding Causal Effects with Instrumental Variables via In-Context Learning](https://mlsanity.com/paper/2605.12924/) (2026)
- [SurvivalPFN: Amortizing Survival Prediction via In-Context Bayesian Inference](https://mlsanity.com/paper/2605.15488/) (2026)
- [FoundCause: Causal Discovery with Latent Confounders from Observational Data](https://mlsanity.com/paper/2606.17516/) (2026)
- [Estimating Time Series Foundation Model Transferability via In-Context Learning](https://mlsanity.com/paper/2509.23695/) (2025)
- [Towards Causal Market Simulators](https://mlsanity.com/paper/2511.04469/) (2025)
- [Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models](https://mlsanity.com/paper/2604.13332/) (2026)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [Computational Identifiability](https://mlsanity.com/paper/2606.19361/) (2026)
- [A Causal Foundation Model for Structure and Outcome Prediction](https://mlsanity.com/paper/2606.26467/) (2026)
- [CDFM: Towards a General-Purpose Causal Discovery Foundation Model](https://mlsanity.com/paper/2607.11508/) (2026)
- [PerturbPFN: Probing the Limits of Synthetic Priors in Drug Perturbation Modelling](https://mlsanity.com/paper/2607.23447/) (2026)
- [DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Series](https://mlsanity.com/paper/2607.27263/) (2026)
- [Understanding the Surprising Generalization Properties of Tabular Foundation Models](https://mlsanity.com/paper/2608.17957/) (2026)
- [Foundation Models for Partial Causal Identification](https://mlsanity.com/paper/2608.20841/) (2026)
- [On the Emergence and Test-Time Use of Structural Information in Large Language Models](https://mlsanity.com/paper/2601.17869/) (2026)

## Abstract (arXiv)

Estimation of causal effects is critical to a range of scientific disciplines. Existing methods for this task either require interventional data, knowledge about the ground truth causal graph, or rely on assumptions such as unconfoundedness, restricting their applicability in real-world settings. In the domain of tabular machine learning, Prior-data fitted networks (PFNs) have achieved state-of-the-art predictive performance, having been pre-trained on synthetic data to solve tabular prediction problems via in-context learning. To assess whether this can be transferred to the harder problem of causal effect estimation, we pre-train PFNs on synthetic data drawn from a wide variety of causal structures, including interventions, to predict interventional outcomes given observational data. Through extensive experiments on synthetic case studies, we show that our approach allows for the accurate estimation of causal effects without knowledge of the underlying causal graph. We also perform ablation studies that elucidate Do-PFN's scalability and robustness across datasets with a variety of causal characteristics.