# Foundation Models for Causal Inference via Prior-Data Fitted Networks

arXiv: https://arxiv.org/abs/2506.10914 · HTML: https://mlsanity.com/paper/2506.10914/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 71 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes](https://mlsanity.com/paper/1704.02801/) (2017)
- [Orthogonal Statistical Learning](https://mlsanity.com/paper/1901.09036/) (2019)
- [Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms](https://mlsanity.com/paper/2101.10943/) (2021)
- [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023)
- [Causal Transformer for Estimating Counterfactual Outcomes](https://mlsanity.com/paper/2204.07258/) (2022)
- [Kernel Instrumental Variable Regression](https://mlsanity.com/paper/1906.00232/) (2019)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Deep Generalized Method of Moments for Instrumental Variable Analysis](https://mlsanity.com/paper/1905.12495/) (2019)
- [Causal machine learning for predicting treatment outcomes](https://mlsanity.com/paper/2410.08770/) (2024)
- [Learning Deep Features in Instrumental Variable Regression](https://mlsanity.com/paper/2010.07154/) (2020)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)
- [Do-PFN: In-Context Learning for Causal Effect Estimation](https://mlsanity.com/paper/2506.06039/) (2025)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects](https://mlsanity.com/paper/1706.09523/) (2017)
- [Causal Effect Identification in Cluster DAGs](https://mlsanity.com/paper/2202.12263/) (2022)
- [Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancing](https://mlsanity.com/paper/2102.04543/) (2021)
- [Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments](https://mlsanity.com/paper/1905.10176/) (2019)
- [Sharp Bounds for Generalized Causal Sensitivity Analysis](https://mlsanity.com/paper/2305.16988/) (2023)
- [Estimating individual treatment effects under unobserved confounding using binary instruments](https://mlsanity.com/paper/2208.08544/) (2022)

## Built on by (most cited first, 30 of 32)

- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [Relational In-Context Learning via Synthetic Pre-training with Structural Prior](https://mlsanity.com/paper/2603.03805/) (2026)
- [Use What You Know: Causal Foundation Models with Partial Graphs](https://mlsanity.com/paper/2602.14972/) (2026)
- [Interventional Time Series Priors for Causal Foundation Models](https://mlsanity.com/paper/2603.11090/) (2026)
- [Does TabPFN Understand Causal Structures?](https://mlsanity.com/paper/2511.07236/) (2025)
- [Causal Foundation Models with Continuous Treatments](https://mlsanity.com/paper/2605.15133/) (2026)
- [Towards Continuous-time Causal Foundation Models](https://mlsanity.com/paper/2605.28880/) (2026)
- [MapPFN: Learning Causal Perturbation Maps in Context](https://mlsanity.com/paper/2601.21092/) (2026)
- [Amortizing Causal Sensitivity Analysis via Prior Data-Fitted Networks](https://mlsanity.com/paper/2605.10590/) (2026)
- [IV-ICL: Bounding Causal Effects with Instrumental Variables via In-Context Learning](https://mlsanity.com/paper/2605.12924/) (2026)
- [LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding](https://mlsanity.com/paper/2507.02843/) (2025)
- [Cluster-Dags as Powerful Background Knowledge For Causal Discovery](https://mlsanity.com/paper/2512.10032/) (2025)
- [Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models](https://mlsanity.com/paper/2604.13332/) (2026)
- [Computational Identifiability](https://mlsanity.com/paper/2606.19361/) (2026)
- [CDFM: Towards a General-Purpose Causal Discovery Foundation Model](https://mlsanity.com/paper/2607.11508/) (2026)
- [PerturbPFN: Probing the Limits of Synthetic Priors in Drug Perturbation Modelling](https://mlsanity.com/paper/2607.23447/) (2026)
- [Foundation Models for Partial Causal Identification](https://mlsanity.com/paper/2608.20841/) (2026)
- [Cloud-Edge Collaborative Large Models for Robust Photovoltaic Power Forecasting](https://mlsanity.com/paper/2603.22343/) (2026)
- [Causal Longitudinal Prior-Fitted Networks for Counterfactual Outcome Prediction](https://mlsanity.com/paper/2606.05797/) (2026)
- [DCD-PFN: A Decoupling-Aware Foundation Model for Causal Discovery](https://mlsanity.com/paper/2606.21212/) (2026)
- [Causal Foundation Models](https://mlsanity.com/paper/2609.03003/) (2026)
- [Integrating a Causal Foundation Model into a Prescriptive Maintenance Framework for Optimising Production-Line OEE](https://mlsanity.com/paper/2512.00969/) (2025)
- [TabSurv: Adapting Modern Tabular Neural Networks to Survival Analysis](https://mlsanity.com/paper/2605.03944/) (2026)
- [TabCF: Distributional Control Function Estimation with Tabular Foundation Models](https://mlsanity.com/paper/2605.05993/) (2026)
- [The Good, the Bad, and the Ugly of Markov Boundary for Tabular Prediction](https://mlsanity.com/paper/2605.29411/) (2026)
- [Temporal Causal Prior-Data Fitted Networks for Panel Data with Learned Reliability Signals](https://mlsanity.com/paper/2606.20889/) (2026)
- [CAFE: Counterfactual Prediction via Fast Posterior Estimation](https://mlsanity.com/paper/2609.32167/) (2026)
- [Can Tabular Foundation Models Amortize Statistical Inference?](https://mlsanity.com/paper/2609.33114/) (2026)
- [What You Observe Determines How You Identify Causal Effects: Evaluating Causal Models across Observational Views](https://mlsanity.com/paper/2609.36881/) (2026)

## Abstract (arXiv)

Prior-data fitted networks (PFNs) have recently been proposed as a promising way to train tabular foundation models. PFNs are transformers that are pre-trained on synthetic data generated from a prespecified prior distribution and that enable Bayesian inference through in-context learning. In this paper, we introduce CausalFM, a comprehensive framework for training PFN-based foundation models in various causal inference settings. First, we formalize the construction of Bayesian priors for causal inference based on structural causal models (SCMs) in a principled way and derive necessary criteria for the validity of such priors. Building on this, we propose a novel family of prior distributions using causality-inspired Bayesian neural networks that enable CausalFM to perform Bayesian causal inference in various settings, including for back-door, front-door, and instrumental variable adjustment. Finally, we instantiate CausalFM and explicitly train models to perform in-context learning in these settings. We show that CausalFM achieves competitive in-context learning performance even when compared to baselines that are specifically trained for the task at hand. In sum, our framework can be used as a general recipe to train foundation models for various causal inference settings. In contrast to the current state-of-the-art in causal inference, CausalFM offers a novel paradigm with the potential to fundamentally change how practitioners perform causal inference in medicine, economics, and other disciplines.