# Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data

arXiv: https://arxiv.org/abs/2512.05456 · HTML: https://mlsanity.com/paper/2512.05456/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 188 references that are in the map.
- First posted on arXiv 5 Dec 2025.

## Builds on (most cited first, 30 of 61)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [Optimal Inference After Model Selection](https://mlsanity.com/paper/1410.2597/) (2014)
- [Using Deep Learning and Google Street View to Estimate the Demographic Makeup of the US](https://mlsanity.com/paper/1702.06683/) (2017)
- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [Cross-Prediction-Powered Inference](https://mlsanity.com/paper/2309.16598/) (2023)
- [Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification](https://mlsanity.com/paper/1909.01940/) (2019)
- [Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models](https://mlsanity.com/paper/2405.02917/) (2024)
- [Active Statistical Inference](https://mlsanity.com/paper/2403.03208/) (2024)
- [Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI](https://mlsanity.com/paper/2501.09731/) (2025)
- [Prediction-Powered Ranking of Large Language Models](https://mlsanity.com/paper/2402.17826/) (2024)
- [Addressing Fairness Issues in Deep Learning-Based Medical Image Analysis: A Systematic Review](https://mlsanity.com/paper/2209.13177/) (2022)
- [Assumption-Lean and Data-Adaptive Post-Prediction Inference](https://mlsanity.com/paper/2311.14220/) (2023)
- [Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation](https://mlsanity.com/paper/2406.04291/) (2024)
- [Prediction-Powered Inference with Imputed Covariates and Nonuniform Sampling](https://mlsanity.com/paper/2501.18577/) (2025)
- [Cost-Optimal Active AI Model Evaluation](https://mlsanity.com/paper/2506.07949/) (2025)
- [Doubly Robust Self-Training](https://mlsanity.com/paper/2306.00265/) (2023)
- [Multi-Label Clinical Time-Series Generation via Conditional GAN](https://mlsanity.com/paper/2204.04797/) (2022)
- [Prediction-Powered E-Values](https://mlsanity.com/paper/2502.04294/) (2025)
- [Tree-Values: selective inference for regression trees](https://mlsanity.com/paper/2106.07816/) (2021)

## Built on by (most cited first, 3 of 3)

- [Spatially Robust Inference with Predicted and Missing at Random Labels](https://mlsanity.com/paper/2603.11368/) (2026)
- [Supercharging Bayesian Inference with Reliable AI-Informed Priors](https://mlsanity.com/paper/2605.09834/) (2026)
- [When Surveys Become Conversations: Adaptive Matrix Validation for AI-Assisted Interviews](https://mlsanity.com/paper/2606.24244/) (2026)

## Abstract (arXiv)

As artificial intelligence and machine learning tools become more accessible, and scientists face new obstacles to data collection (e.g., rising costs, declining survey response rates), researchers increasingly use predictions from pre-trained algorithms as substitutes for missing or unobserved data. Though appealing for financial and logistical reasons, using standard tools for inference can misrepresent the association between independent variables and the outcome of interest when the true, unobserved outcome is replaced by a predicted value. In this paper, we characterize the statistical challenges inherent to drawing inference with predicted data (IPD) and show that high predictive accuracy does not guarantee valid downstream inference. We show that all such failures reduce to statistical notions of (i) bias, when predictions systematically shift the estimand or distort relationships among variables, and (ii) variance, when uncertainty from the prediction model and the intrinsic variability of the true data are ignored. We then review recent methods for conducting IPD and discuss how this framework is deeply rooted in classical statistical theory. We then comment on some open questions and interesting avenues for future work in this area, and end with some comments on how to use predicted data in scientific studies that is both transparent and statistically principled.