# DoubleMLDeep: Estimation of Causal Effects with Multimodal Data

arXiv: https://arxiv.org/abs/2402.01785 · HTML: https://mlsanity.com/paper/2402.01785/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 38 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 12 of 12)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Orthogonal Statistical Learning](https://mlsanity.com/paper/1901.09036/) (2019)
- [Deep Neural Networks for Estimation and Inference](https://mlsanity.com/paper/1809.09953/) (2018)
- [How to Make Causal Inferences Using Texts](https://mlsanity.com/paper/1802.02163/) (2018)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [DoubleML -- An Object-Oriented Implementation of Double Machine Learning in Python](https://mlsanity.com/paper/2104.03220/) (2021)
- [DoubleML -- An Object-Oriented Implementation of Double Machine Learning in R](https://mlsanity.com/paper/2103.09603/) (2021)

## Built on by (most cited first, 12 of 12)

- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Applied Causal Inference Powered by ML and AI](https://mlsanity.com/paper/2403.02467/) (2024)
- [Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatments](https://mlsanity.com/paper/2410.00903/) (2024)
- [Adjustment for Confounding using Pre-Trained Representations](https://mlsanity.com/paper/2506.14329/) (2025)
- [Advancing Precision Oncology Through Modeling of Longitudinal and Multimodal Data](https://mlsanity.com/paper/2502.07836/) (2025)
- [Towards Representation Learning for Weighting Problems in Design-Based Causal Inference](https://mlsanity.com/paper/2409.16407/) (2024)
- [Leveraging Generative Artificial Intelligence for Causal Inference with Unstructured Data](https://mlsanity.com/paper/2507.03897/) (2025)
- [Causal Inference with Unstructured Treatments](https://mlsanity.com/paper/2608.00657/) (2026)
- [Annotation-Assisted Learning of Treatment Policies From Multimodal Electronic Health Records](https://mlsanity.com/paper/2507.20993/) (2025)
- [SHIFT: Robust Double Machine Learning for Average Dose-Response Functions under Heavy-Tailed Contamination](https://mlsanity.com/paper/2605.00176/) (2026)
- [Econometrics with Pre-Trained Embeddings for Unstructured Data](https://mlsanity.com/paper/2607.17378/) (2026)
- [Causal Inference with Unstructured Outcomes](https://mlsanity.com/paper/2608.03085/) (2026)

## Abstract (arXiv)

This paper explores the use of unstructured, multimodal data, namely text and images, in causal inference and treatment effect estimation. We propose a neural network architecture that is adapted to the double machine learning (DML) framework, specifically the partially linear model. An additional contribution of our paper is a new method to generate a semi-synthetic dataset which can be used to evaluate the performance of causal effect estimation in the presence of text and images as confounders. The proposed methods and architectures are evaluated on the semi-synthetic dataset and compared to standard approaches, highlighting the potential benefit of using text and images directly in causal studies. Our findings have implications for researchers and practitioners in economics, marketing, finance, medicine and data science in general who are interested in estimating causal quantities using non-traditional data.