# Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies

arXiv: https://arxiv.org/abs/2203.09672 · HTML: https://mlsanity.com/paper/2203.09672/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 70 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [The Blessings of Multiple Causes](https://mlsanity.com/paper/1805.06826/) (2018)
- [The Causal-Neural Connection: Expressiveness, Learnability, and Inference](https://mlsanity.com/paper/2107.00793/) (2021)
- [Support and Invertibility in Domain-Invariant Representations](https://mlsanity.com/paper/1903.03448/) (2019)
- [CLIP-Event: Connecting Text and Images with Event Structures](https://mlsanity.com/paper/2201.05078/) (2022)
- [Treatment effect estimation with disentangled latent factors](https://mlsanity.com/paper/2001.10652/) (2020)
- [How to Make Causal Inferences Using Texts](https://mlsanity.com/paper/1802.02163/) (2018)
- [Counterfactual Fairness with Disentangled Causal Effect Variational Autoencoder](https://mlsanity.com/paper/2011.11878/) (2020)
- [DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Training](https://mlsanity.com/paper/1802.05664/) (2018)
- [Learning Weighted Representations for Generalization Across Designs](https://mlsanity.com/paper/1802.08598/) (2018)
- [Causal Effects of Linguistic Properties](https://mlsanity.com/paper/2010.12919/) (2020)
- [Using Embeddings to Correct for Unobserved Confounding in Networks](https://mlsanity.com/paper/1902.04114/) (2019)
- [Implicit Causal Models for Genome-wide Association Studies](https://mlsanity.com/paper/1710.10742/) (2017)
- [MissDeepCausal: Causal Inference from Incomplete Data Using Deep Latent Variable Models](https://mlsanity.com/paper/2002.10837/) (2020)
- [Targeted VAE: Variational and Targeted Learning for Causal Inference](https://mlsanity.com/paper/2009.13472/) (2020)
- [Counterfactual Prediction with Deep Instrumental Variables Networks](https://mlsanity.com/paper/1612.09596/) (2016)
- [Towards Principled Causal Effect Estimation by Deep Identifiable Models](https://mlsanity.com/paper/2109.15062/) (2021)

## Built on by (most cited first, 13 of 13)

- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Diffusion Models With Learned Adaptive Noise](https://mlsanity.com/paper/2312.13236/) (2023)
- [ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation](https://mlsanity.com/paper/2311.13258/) (2023)
- [Deep Causal Learning: Representation, Discovery and Inference](https://mlsanity.com/paper/2211.03374/) (2022)
- [Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources](https://mlsanity.com/paper/2305.06217/) (2023)
- [Estimating Causal Effects using a Multi-task Deep Ensemble](https://mlsanity.com/paper/2301.11351/) (2023)
- [Towards the Causal Complete Cause of Multi-Modal Representation Learning](https://mlsanity.com/paper/2407.14058/) (2024)
- [Calibrated and Conformal Propensity Scores for Causal Effect Estimation](https://mlsanity.com/paper/2306.00382/) (2023)
- [Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors](https://mlsanity.com/paper/2401.02739/) (2024)
- [LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding](https://mlsanity.com/paper/2507.02843/) (2025)
- [Recovering Latent Confounders from High-dimensional Proxy Variables](https://mlsanity.com/paper/2403.14228/) (2024)
- [Causal Prompt Calibration Guided Segment Anything Model for Open-Vocabulary Multi-Entity Segmentation](https://mlsanity.com/paper/2505.06524/) (2025)
- [Robust Multimodal Representation Learning in Healthcare](https://mlsanity.com/paper/2601.21941/) (2026)

## Abstract (arXiv)

Estimating the effect of intervention from observational data while accounting for confounding variables is a key task in causal inference. Oftentimes, the confounders are unobserved, but we have access to large amounts of additional unstructured data (images, text) that contain valuable proxy signal about the missing confounders. This paper argues that leveraging this unstructured data can greatly improve the accuracy of causal effect estimation. Specifically, we introduce deep multi-modal structural equations, a generative model for causal effect estimation in which confounders are latent variables and unstructured data are proxy variables. This model supports multiple multi-modal proxies (images, text) as well as missing data. We empirically demonstrate that our approach outperforms existing methods based on propensity scores and corrects for confounding using unstructured inputs on tasks in genomics and healthcare. Our methods can potentially support the use of large amounts of data that were previously not used in causal inference