# Adjustment for Confounding using Pre-Trained Representations

arXiv: https://arxiv.org/abs/2506.14329 · HTML: https://mlsanity.com/paper/2506.14329/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 66 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder](https://mlsanity.com/paper/1609.08816/) (2016)
- [TorchXRayVision: A library of chest X-ray datasets and models](https://mlsanity.com/paper/2111.00595/) (2021)
- [Causal Transformer for Estimating Counterfactual Outcomes](https://mlsanity.com/paper/2204.07258/) (2022)
- [Double/Debiased/Neyman Machine Learning of Treatment Effects](https://mlsanity.com/paper/1701.08687/) (2017)
- [Deep ReLU network approximation of functions on a manifold](https://mlsanity.com/paper/1908.00695/) (2019)
- [CausalML: Python Package for Causal Machine Learning](https://mlsanity.com/paper/2002.11631/) (2020)
- [On the limits of cross-domain generalization in automated X-ray prediction](https://mlsanity.com/paper/2002.02497/) (2020)
- [On the Intrinsic Dimensionality of Image Representations](https://mlsanity.com/paper/1803.09672/) (2018)
- [Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction](https://mlsanity.com/paper/2105.04544/) (2021)
- [Using Embeddings to Correct for Unobserved Confounding in Networks](https://mlsanity.com/paper/1902.04114/) (2019)
- [Causal Inference with Noisy and Missing Covariates via Matrix Factorization](https://mlsanity.com/paper/1806.00811/) (2018)
- [DoubleML -- An Object-Oriented Implementation of Double Machine Learning in Python](https://mlsanity.com/paper/2104.03220/) (2021)
- [Image-based Treatment Effect Heterogeneity](https://mlsanity.com/paper/2206.06417/) (2022)
- [End-To-End Causal Effect Estimation from Unstructured Natural Language Data](https://mlsanity.com/paper/2407.07018/) (2024)
- [The Effect of Intrinsic Dataset Properties on Generalization: Unraveling Learning Differences Between Natural and Medical Images](https://mlsanity.com/paper/2401.08865/) (2024)
- [DoubleMLDeep: Estimation of Causal Effects with Multimodal Data](https://mlsanity.com/paper/2402.01785/) (2024)
- [Integrating Earth Observation Data into Causal Inference: Challenges and Opportunities](https://mlsanity.com/paper/2301.12985/) (2023)
- [Estimation of a regression function on a manifold by fully connected deep neural networks](https://mlsanity.com/paper/2107.09532/) (2021)
- [Efficient adjustment for complex covariates: Gaining efficiency with DOPE](https://mlsanity.com/paper/2402.12980/) (2024)
- [Estimating Causal Effects Under Image Confounding Bias with an Application to Poverty in Africa](https://mlsanity.com/paper/2206.06410/) (2022)
- [Causal inference through multi-stage learning and doubly robust deep neural networks](https://mlsanity.com/paper/2407.08560/) (2024)

## Built on by (most cited first, 4 of 4)

- [GDR-learners: Orthogonal Learning of Generative Models for Potential Outcomes](https://mlsanity.com/paper/2509.22953/) (2025)
- [Rethinking Intrinsic Dimension Estimation in Neural Representations](https://mlsanity.com/paper/2604.20276/) (2026)
- [Econometrics with Pre-Trained Embeddings for Unstructured Data](https://mlsanity.com/paper/2607.17378/) (2026)
- [Exploring Sparse Autoencoders in Text-Based Causal Confounding Adjustment](https://mlsanity.com/paper/2609.01322/) (2026)

## Abstract (arXiv)

There is growing interest in extending average treatment effect (ATE) estimation to incorporate non-tabular data, such as images and text, which may act as sources of confounding. Neglecting these effects risks biased results and flawed scientific conclusions. However, incorporating non-tabular data necessitates sophisticated feature extractors, often in combination with ideas of transfer learning. In this work, we investigate how latent features from pre-trained neural networks can be leveraged to adjust for sources of confounding. We formalize conditions under which these latent features enable valid adjustment and statistical inference in ATE estimation, demonstrating results along the example of double machine learning. We discuss critical challenges inherent to latent feature learning and downstream parameter estimation arising from the high dimensionality and non-identifiability of representations. Common structural assumptions for obtaining fast convergence rates with additive or sparse linear models are shown to be unrealistic for latent features. We argue, however, that neural networks are largely insensitive to these issues. In particular, we show that neural networks can achieve fast convergence rates by adapting to intrinsic notions of sparsity and dimension of the learning problem.