# No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference

arXiv: https://arxiv.org/abs/2505.20178 · HTML: https://mlsanity.com/paper/2505.20178/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [The price of debiasing automatic metrics in natural language evaluation](https://mlsanity.com/paper/1807.02202/) (2018)
- [Cross-Prediction-Powered Inference](https://mlsanity.com/paper/2309.16598/) (2023)
- [AutoEval Done Right: Using Synthetic Data for Model Evaluation](https://mlsanity.com/paper/2403.07008/) (2024)
- [Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data](https://mlsanity.com/paper/2410.13341/) (2024)
- [Active Statistical Inference](https://mlsanity.com/paper/2403.03208/) (2024)
- [Can Unconfident LLM Annotations Be Used for Confident Conclusions?](https://mlsanity.com/paper/2408.15204/) (2024)
- [Elastic Integrative Analysis of Randomized Trial and Real-World Data for Treatment Heterogeneity Estimation](https://mlsanity.com/paper/2005.10579/) (2020)
- [Prediction-powered Generalization of Causal Inferences](https://mlsanity.com/paper/2406.02873/) (2024)
- [Minimax Rates and Adaptivity in Combining Experimental and Observational Data](https://mlsanity.com/paper/2109.10522/) (2021)
- [Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression](https://mlsanity.com/paper/2411.12665/) (2024)
- [Prediction-Powered Adaptive Shrinkage Estimation](https://mlsanity.com/paper/2502.14166/) (2025)
- [Understanding the Risks and Rewards of Combining Unbiased and Possibly Biased Estimators, with Applications to Causal Inference](https://mlsanity.com/paper/2205.10467/) (2022)

## Built on by (most cited first, 8 of 8)

- [How Benchmark Prediction from Fewer Data Misses the Mark](https://mlsanity.com/paper/2506.07673/) (2025)
- [Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification](https://mlsanity.com/paper/2510.11408/) (2025)
- [Demystifying Prediction Powered Inference](https://mlsanity.com/paper/2601.20819/) (2026)
- [Revisiting Active Sequential Prediction-Powered Mean Estimation](https://mlsanity.com/paper/2604.18569/) (2026)
- [Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data](https://mlsanity.com/paper/2512.05456/) (2025)
- [Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation](https://mlsanity.com/paper/2608.26638/) (2026)
- [ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control](https://mlsanity.com/paper/2609.34374/) (2026)
- [How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats](https://mlsanity.com/paper/2609.35815/) (2026)

## Abstract (arXiv)

Prediction-Powered Inference (PPI) is a popular strategy for combining gold-standard and possibly noisy pseudo-labels to perform statistical estimation. Prior work has shown an asymptotic enquotefree lunch for PPI++, an adaptive form of PPI, showing that the textitasymptotic variance of PPI++ is always less than or equal to the variance obtained from using gold-standard labels alone. Notably, this result holds textitregardless of the quality of the pseudo-labels. In this work, we demystify this result by conducting an exact finite-sample analysis of the estimation error of PPI++ on the mean estimation problem. We give a enquoteno free lunch result, characterizing the settings (and sample sizes) where PPI++ has provably worse estimation error than using gold-standard labels alone. Specifically, PPI++ will outperform if and only if the correlation between pseudo- and gold-standard is above a certain level that depends on the number of labeled samples (n). In some cases our results simplify considerably: For Gaussian data, for instance, the correlation must be at least 1/√n - 2 in order to see improvement. More broadly, by providing exact non-asymptotic expressions for the variance of PPI++ under sample splitting, we aim to empower practitioners to transparently reason about the benefits of PPI++ in specific applications. In experiments, we illustrate that our theoretical findings hold on real-world datasets.