# The Causal-Neural Connection: Expressiveness, Learnability, and Inference

arXiv: https://arxiv.org/abs/2107.00793 · HTML: https://mlsanity.com/paper/2107.00793/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 75 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation](https://mlsanity.com/paper/1602.03253/) (2016)
- [Differentiable Causal Discovery from Interventional Data](https://mlsanity.com/paper/2007.01754/) (2020)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training](https://mlsanity.com/paper/1709.02023/) (2017)
- [Generalization Bounds and Representation Learning for Estimation of Potential Outcomes and Causal Effects](https://mlsanity.com/paper/2001.07426/) (2020)
- [Counterfactual Probabilities: Computational Methods, Bounds and Applications](https://mlsanity.com/paper/1302.6784/) (2013)
- [A Framework For Contrastive Self-Supervised Learning And Designing A New Approach](https://mlsanity.com/paper/2009.00104/) (2020)
- [DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Training](https://mlsanity.com/paper/1802.05664/) (2018)
- [Semiparametric counterfactual density estimation](https://mlsanity.com/paper/2102.12034/) (2021)
- [A Survey of Learning Causality with Data: Problems and Methods](https://mlsanity.com/paper/1809.09337/) (2018)
- [Adversarial Balancing-based Representation Learning for Causal Effect Inference with Observational Data](https://mlsanity.com/paper/1904.13335/) (2019)
- [Causal Identification under Markov Equivalence](https://mlsanity.com/paper/1812.06209/) (2018)

## Built on by (most cited first, 30 of 98)

- [Causal Parrots: Large Language Models May Talk Causality But Are Not Causal](https://mlsanity.com/paper/2308.13067/) (2023)
- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [Neural Causal Models for Counterfactual Identification and Estimation](https://mlsanity.com/paper/2210.00035/) (2022)
- [High Fidelity Image Counterfactuals with Probabilistic Causal Models](https://mlsanity.com/paper/2306.15764/) (2023)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [Counterfactual Identifiability of Bijective Causal Models](https://mlsanity.com/paper/2302.02228/) (2023)
- [Relating Graph Neural Networks to Structural Causal Models](https://mlsanity.com/paper/2109.04173/) (2021)
- [Learning Neural Causal Models with Active Interventions](https://mlsanity.com/paper/2109.02429/) (2021)
- [Defining and Quantifying the Emergence of Sparse Concepts in DNNs](https://mlsanity.com/paper/2111.06206/) (2021)
- [Causal Interpretation of Self-Attention in Pre-Trained Transformers](https://mlsanity.com/paper/2310.20307/) (2023)
- [Sharp Bounds for Generalized Causal Sensitivity Analysis](https://mlsanity.com/paper/2305.16988/) (2023)
- [Causal Transportability for Visual Recognition](https://mlsanity.com/paper/2204.12363/) (2022)
- [Passive learning of active causal strategies in agents and language models](https://mlsanity.com/paper/2305.16183/) (2023)
- [Neural Causal Abstractions](https://mlsanity.com/paper/2401.02602/) (2024)
- [Counterfactual Image Editing](https://mlsanity.com/paper/2403.09683/) (2024)
- [VACA: Design of Variational Graph Autoencoders for Interventional and Counterfactual Queries](https://mlsanity.com/paper/2110.14690/) (2021)
- [Partial Identification of Treatment Effects with Implicit Generative Models](https://mlsanity.com/paper/2210.08139/) (2022)
- [Counterfactual (Non-)identifiability of Learned Structural Causal Models](https://mlsanity.com/paper/2301.09031/) (2023)
- [Towards Causal Foundation Model: on Duality between Causal Inference and Attention](https://mlsanity.com/paper/2310.00809/) (2023)
- [A Neural Framework for Generalized Causal Sensitivity Analysis](https://mlsanity.com/paper/2311.16026/) (2023)
- [Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Model](https://mlsanity.com/paper/2306.01424/) (2023)
- [Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies](https://mlsanity.com/paper/2203.09672/) (2022)
- [A Review of Change of Variable Formulas for Generative Modeling](https://mlsanity.com/paper/2308.02652/) (2023)
- [Hybrid2 Neural ODE Causal Modeling and an Application to Glycemic Response](https://mlsanity.com/paper/2402.17233/) (2024)
- [Black Box Causal Inference: Effect Estimation via Meta Prediction](https://mlsanity.com/paper/2503.05985/) (2025)
- [Unicorn: Reasoning about Configurable System Performance through the lens of Causality](https://mlsanity.com/paper/2201.08413/) (2022)
- [Mitigating attribute amplification in counterfactual image generation](https://mlsanity.com/paper/2403.09422/) (2024)
- [A Meta-Reinforcement Learning Algorithm for Causal Discovery](https://mlsanity.com/paper/2207.08457/) (2022)
- [Deep Causal Learning: Representation, Discovery and Inference](https://mlsanity.com/paper/2211.03374/) (2022)
- [Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference](https://mlsanity.com/paper/2401.01426/) (2024)

## Abstract (arXiv)

One of the central elements of any causal inference is an object called structural causal model (SCM), which represents a collection of mechanisms and exogenous sources of random variation of the system under investigation (Pearl, 2000). An important property of many kinds of neural networks is universal approximability: the ability to approximate any function to arbitrary precision. Given this property, one may be tempted to surmise that a collection of neural nets is capable of learning any SCM by training on data generated by that SCM. In this paper, we show this is not the case by disentangling the notions of expressivity and learnability. Specifically, we show that the causal hierarchy theorem (Thm. 1, Bareinboim et al., 2020), which describes the limits of what can be learned from data, still holds for neural models. For instance, an arbitrarily complex and expressive neural net is unable to predict the effects of interventions given observational data alone. Given this result, we introduce a special type of SCM called a neural causal model (NCM), and formalize a new type of inductive bias to encode structural constraints necessary for performing causal inferences. Building on this new class of models, we focus on solving two canonical tasks found in the literature known as causal identification and estimation. Leveraging the neural toolbox, we develop an algorithm that is both sufficient and necessary to determine whether a causal effect can be learned from data (i.e., causal identifiability); it then estimates the effect whenever identifiability holds (causal estimation). Simulations corroborate the proposed approach.