# Active Statistical Inference

arXiv: https://arxiv.org/abs/2403.03208 · HTML: https://mlsanity.com/paper/2403.03208/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 59 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 23 of 23)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [Confidence Intervals for Policy Evaluation in Adaptive Experiments](https://mlsanity.com/paper/1911.02768/) (2019)
- [A Generalizable and Accessible Approach to Machine Learning with Global Satellite Imagery](https://mlsanity.com/paper/2010.08168/) (2020)
- [Cross-Prediction-Powered Inference](https://mlsanity.com/paper/2309.16598/) (2023)
- [Tight Concentrations and Confidence Sequences from the Regret of Universal Portfolio](https://mlsanity.com/paper/2110.14099/) (2021)
- [Statistical Inference with M-Estimators on Adaptively Collected Data](https://mlsanity.com/paper/2104.14074/) (2021)
- [Semi-supervised Inference: General Theory and Estimation of Means](https://mlsanity.com/paper/1606.07268/) (2016)
- [Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping](https://mlsanity.com/paper/1510.00098/) (2015)
- [Active Learning for Optimal Intervention Design in Causal Models](https://mlsanity.com/paper/2209.04744/) (2022)
- [Assumption-Lean and Data-Adaptive Post-Prediction Inference](https://mlsanity.com/paper/2311.14220/) (2023)
- [Efficient Adaptive Experimental Design for Average Treatment Effect Estimation](https://mlsanity.com/paper/2002.05308/) (2020)
- [Semiparametric Efficient Inference in Adaptive Experiments](https://mlsanity.com/paper/2311.18274/) (2023)
- [Learning Economic Parameters from Revealed Preferences](https://mlsanity.com/paper/1407.7937/) (2014)
- [How many labelers do you have? A closer look at gold-standard labels](https://mlsanity.com/paper/2206.12041/) (2022)
- [Promises and Pitfalls of Threshold-based Auto-labeling](https://mlsanity.com/paper/2211.12620/) (2022)
- [Adaptive Instrument Design for Indirect Experiments](https://mlsanity.com/paper/2312.02438/) (2023)

## Built on by (most cited first, 30 of 50)

- [Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI](https://mlsanity.com/paper/2501.09731/) (2025)
- [Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation](https://mlsanity.com/paper/2406.04291/) (2024)
- [Cost-Optimal Active AI Model Evaluation](https://mlsanity.com/paper/2506.07949/) (2025)
- [Efficient Evaluation of LLM Performance with Statistical Guarantees](https://mlsanity.com/paper/2601.20251/) (2026)
- [A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation](https://mlsanity.com/paper/2406.07320/) (2024)
- [Active, anytime-valid risk controlling prediction sets](https://mlsanity.com/paper/2406.10490/) (2024)
- [Multi-Armed Bandits With Machine Learning-Generated Surrogate Rewards](https://mlsanity.com/paper/2506.16658/) (2025)
- [Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators](https://mlsanity.com/paper/2510.04354/) (2025)
- [Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees](https://mlsanity.com/paper/2505.18659/) (2025)
- [Sequential Harmful Shift Detection Without Labels](https://mlsanity.com/paper/2412.12910/) (2024)
- [Best Arm Identification with LLM Judges and Limited Human](https://mlsanity.com/paper/2601.21471/) (2026)
- [Demystifying Prediction Powered Inference](https://mlsanity.com/paper/2601.20819/) (2026)
- [No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference](https://mlsanity.com/paper/2505.20178/) (2025)
- [Probably Approximately Correct Labels](https://mlsanity.com/paper/2506.10908/) (2025)
- [Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression](https://mlsanity.com/paper/2411.12665/) (2024)
- [Another look at statistical inference with machine learning-imputed data](https://mlsanity.com/paper/2411.19908/) (2024)
- [FAB-PPI: Frequentist, Assisted by Bayes, Prediction-Powered Inference](https://mlsanity.com/paper/2502.02363/) (2025)
- [Anytime-valid, Bayes-assisted, Prediction-Powered Inference](https://mlsanity.com/paper/2505.18000/) (2025)
- [Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking](https://mlsanity.com/paper/2605.05973/) (2026)
- [Robust Sampling for Active Statistical Inference](https://mlsanity.com/paper/2511.08991/) (2025)
- [Adaptive Off-Policy Inference for M-Estimators Under Model Misspecification](https://mlsanity.com/paper/2509.14218/) (2025)
- [ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking](https://mlsanity.com/paper/2511.09833/) (2025)
- [PPI-SVRG: Unifying Prediction-Powered Inference and Variance Reduction for Semi-Supervised Optimization](https://mlsanity.com/paper/2601.21470/) (2026)
- [Revisiting Active Sequential Prediction-Powered Mean Estimation](https://mlsanity.com/paper/2604.18569/) (2026)
- [Calibeating Prediction-Powered Inference](https://mlsanity.com/paper/2604.21260/) (2026)
- [High-Dimensional Statistics: Reflections on Progress and Open Problems](https://mlsanity.com/paper/2605.05076/) (2026)
- [Efficient Sequential Evaluation of Large Language Models](https://mlsanity.com/paper/2607.17409/) (2026)
- [Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMs](https://mlsanity.com/paper/2506.13593/) (2025)
- [Model-agnostic Selective Labeling with Provable Statistical Guarantees](https://mlsanity.com/paper/2510.14581/) (2025)
- [Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data](https://mlsanity.com/paper/2512.05456/) (2025)

## Abstract (arXiv)

Inspired by the concept of active learning, we propose active inferenceunicodex2013a methodology for statistical inference with machine-learning-assisted data collection. Assuming a budget on the number of labels that can be collected, the methodology uses a machine learning model to identify which data points would be most beneficial to label, thus effectively utilizing the budget. It operates on a simple yet powerful intuition: prioritize the collection of labels for data points where the model exhibits uncertainty, and rely on the model's predictions where it is confident. Active inference constructs provably valid confidence intervals and hypothesis tests while leveraging any black-box machine learning model and handling any data distribution. The key point is that it achieves the same level of accuracy with far fewer samples than existing baselines relying on non-adaptively-collected data. This means that for the same number of collected samples, active inference enables smaller confidence intervals and more powerful p-values. We evaluate active inference on datasets from public opinion research, census analysis, and proteomics.