# Robust Sampling for Active Statistical Inference

arXiv: https://arxiv.org/abs/2511.08991 · HTML: https://mlsanity.com/paper/2511.08991/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 27 of 27)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [A Computational Approach to Politeness with Application to Social Factors](https://mlsanity.com/paper/1306.6078/) (2013)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models](https://mlsanity.com/paper/2302.13439/) (2023)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Cross-Prediction-Powered Inference](https://mlsanity.com/paper/2309.16598/) (2023)
- [Active Statistical Inference](https://mlsanity.com/paper/2403.03208/) (2024)
- [Can Unconfident LLM Annotations Be Used for Confident Conclusions?](https://mlsanity.com/paper/2408.15204/) (2024)
- [Calibrating the Confidence of Large Language Models by Eliciting Fidelity](https://mlsanity.com/paper/2404.02655/) (2024)
- [Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models](https://mlsanity.com/paper/2502.11028/) (2025)
- [Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping](https://mlsanity.com/paper/1510.00098/) (2015)
- [Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI](https://mlsanity.com/paper/2501.09731/) (2025)
- [Assumption-Lean and Data-Adaptive Post-Prediction Inference](https://mlsanity.com/paper/2311.14220/) (2023)
- [Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation](https://mlsanity.com/paper/2406.04291/) (2024)
- [Prediction-Powered Inference with Imputed Covariates and Nonuniform Sampling](https://mlsanity.com/paper/2501.18577/) (2025)
- [Principal Geodesic Analysis for Probability Measures under the Optimal Transport Metric](https://mlsanity.com/paper/1506.07944/) (2015)
- [Prediction De-Correlated Inference: A safe approach for post-prediction inference](https://mlsanity.com/paper/2312.06478/) (2023)
- [A Note on the Prediction-Powered Bootstrap](https://mlsanity.com/paper/2405.18379/) (2024)
- [Task-Agnostic Machine-Learning-Assisted Inference](https://mlsanity.com/paper/2405.20039/) (2024)
- [Prediction-Guided Active Experiments](https://mlsanity.com/paper/2411.12036/) (2024)
- [Robust online active learning](https://mlsanity.com/paper/2302.00422/) (2023)
- [Robust Active Learning: Sample-Efficient Training of Robust Deep Learning Models](https://mlsanity.com/paper/2112.02542/) (2021)

## Built on by (most cited first, 6 of 6)

- [Active Multiple-Prediction-Powered Inference](https://mlsanity.com/paper/2605.08429/) (2026)
- [Optimized Labeling Resource Allocation for Prediction-Assisted Inference via OPAL](https://mlsanity.com/paper/2606.03211/) (2026)
- [Learning U-Statistics with Active Inference](https://mlsanity.com/paper/2605.11638/) (2026)
- [BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges](https://mlsanity.com/paper/2607.16239/) (2026)
- [Human-Anchored Factuality Evaluation with Strategic Annotation](https://mlsanity.com/paper/2609.00494/) (2026)
- [The Delegation Blind Spot: Auditing Product Decisions from Agent Choices](https://mlsanity.com/paper/2609.26642/) (2026)

## Abstract (arXiv)

Active statistical inference is a new method for inference with AI-assisted data collection. Given a budget on the number of labeled data points that can be collected and assuming access to an AI predictive model, the basic idea is to improve estimation accuracy by prioritizing the collection of labels where the model is most uncertain. The drawback, however, is that inaccurate uncertainty estimates can make active sampling produce highly noisy results, potentially worse than those from naive uniform sampling. In this work, we present robust sampling strategies for active statistical inference. Robust sampling ensures that the resulting estimator is never worse than the estimator using uniform sampling. Furthermore, with reliable uncertainty estimates, the estimator usually outperforms standard active inference. This is achieved by optimally interpolating between uniform and active sampling, depending on the quality of the uncertainty scores, and by using ideas from robust optimization. We demonstrate the utility of the method on a series of real datasets from computational social science and survey research.