# The Hardness of Validating Observational Studies with Experimental Data

arXiv: https://arxiv.org/abs/2503.14795 · HTML: https://mlsanity.com/paper/2503.14795/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Builds on (most cited first, 18 of 18)

- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes](https://mlsanity.com/paper/1704.02801/) (2017)
- [Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms](https://mlsanity.com/paper/2101.10943/) (2021)
- [Removing Hidden Confounding by Experimental Grounding](https://mlsanity.com/paper/1810.11646/) (2018)
- [Uniform Error Bounds for Gaussian Process Regression with Application to Safe Control](https://mlsanity.com/paper/1906.01376/) (2019)
- [Causal inference methods for combining randomized trials and observational studies: a review](https://mlsanity.com/paper/2011.08047/) (2020)
- [Practical and Rigorous Uncertainty Bounds for Gaussian Process Regression](https://mlsanity.com/paper/2105.02796/) (2021)
- [Kernel Conditional Moment Test via Maximum Moment Restriction](https://mlsanity.com/paper/2002.09225/) (2020)
- [Estimating long-term causal effects from short-term experiments and long-term observational data with unobserved confounding](https://mlsanity.com/paper/2302.10625/) (2023)
- [Hypothesis Testing For Densities and High-Dimensional Multinomials: Sharp Local Minimax Rates](https://mlsanity.com/paper/1706.10003/) (2017)
- [Falsification before Extrapolation in Causal Effect Estimation](https://mlsanity.com/paper/2209.13708/) (2022)
- [Disentangling Causal Effects from Sets of Interventions in the Presence of Unobserved Confounders](https://mlsanity.com/paper/2210.05446/) (2022)
- [Falsification of Internal and External Validity in Observational Studies via Conditional Moment Restrictions](https://mlsanity.com/paper/2301.13133/) (2023)
- [Leveraging directed causal discovery to detect latent common causes](https://mlsanity.com/paper/1910.10174/) (2019)
- [Hidden yet quantifiable: A lower bound for confounding strength using randomized trials](https://mlsanity.com/paper/2312.03871/) (2023)
- [Detecting critical treatment effect bias in small subgroups](https://mlsanity.com/paper/2404.18905/) (2024)
- [Nonparametric identification is not enough, but randomized controlled trials are](https://mlsanity.com/paper/2108.11342/) (2021)
- [Impossible Inference in Econometrics: Theory and Applications](https://mlsanity.com/paper/1612.02024/) (2016)

## Built on by (most cited first, 4 of 4)

- [Local Interference: Removing Interference Bias in Semi-Parametric Causal Models](https://mlsanity.com/paper/2503.18756/) (2025)
- [Robust estimation of heterogeneous treatment effects in randomized trials leveraging external data](https://mlsanity.com/paper/2507.03681/) (2025)
- [Observationally Informed Adaptive Causal Experimental Design](https://mlsanity.com/paper/2603.03785/) (2026)
- [Uncovering Bias Mechanisms in Observational Studies](https://mlsanity.com/paper/2506.01191/) (2025)

## Abstract (arXiv)

Observational data is often readily available in large quantities, but can lead to biased causal effect estimates due to the presence of unobserved confounding. Recent works attempt to remove this bias by supplementing observational data with experimental data, which, when available, is typically on a smaller scale due to the time and cost involved in running a randomised controlled trial. In this work, we prove a theorem that places fundamental limits on this ``best of both worlds'' approach. Using the framework of impossible inference, we show that although it is possible to use experimental data to falsify causal effect estimates from observational data, in general it is not possible to validate such estimates. Our theorem proves that while experimental data can be used to detect bias in observational studies, without additional assumptions on the smoothness of the correction function, it can not be used to remove it. We provide a practical example of such an assumption, developing a novel Gaussian Process based approach to construct intervals which contain the true treatment effect with high probability, both inside and outside of the support of the experimental data. We demonstrate our methodology on both simulated and semi-synthetic datasets and make the hrefhttps://github.com/Jakefawkes/Obsandexpdatacode available.