# Semi-Supervised Approaches to Efficient Evaluation of Model Prediction Performance

arXiv: https://arxiv.org/abs/1711.05663 · HTML: https://mlsanity.com/paper/1711.05663/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2017.

## Built on by (most cited first, 9 of 9)

- [Semi-Supervised Quantile Estimation: Robust and Efficient Inference in High Dimensional Settings](https://mlsanity.com/paper/2201.10208/) (2022)
- [SUDO: a framework for evaluating clinical artificial intelligence systems without ground-truth annotations](https://mlsanity.com/paper/2403.17011/) (2024)
- [Semi-supervised Batch Learning From Logged Data](https://mlsanity.com/paper/2209.07148/) (2022)
- [Distributed Semi-Supervised Sparse Statistical Inference](https://mlsanity.com/paper/2306.10395/) (2023)
- [On semi-supervised estimation using exponential tilt mixture models](https://mlsanity.com/paper/2311.08504/) (2023)
- [Efficient semi-supervised inference for logistic regression under case-control studies](https://mlsanity.com/paper/2402.15365/) (2024)
- [Latent Factor Point Processes for Patient Representation in Electronic Health Records](https://mlsanity.com/paper/2508.20327/) (2025)
- [Inferential Evaluation of Surrogate-Derived Models under Covariate Shift](https://mlsanity.com/paper/2608.15783/) (2026)
- [Targeted Review for AI-Assisted Biodiversity Surveys: Active Continuous-Score Occupancy Modeling](https://mlsanity.com/paper/2609.25657/) (2026)

## Abstract (arXiv)

In many modern machine learning applications, the outcome is expensive or time-consuming to collect while the predictor information is easy to obtain. Semi-supervised learning (SSL) aims at utilizing large amounts of `unlabeled' data along with small amounts of `labeled' data to improve the efficiency of a classical supervised approach. Though numerous SSL classification and prediction procedures have been proposed in recent years, no methods currently exist to evaluate the prediction performance of a working regression model. In the context of developing phenotyping algorithms derived from electronic medical records (EMR), we present an efficient two-step estimation procedure for evaluating a binary classifier based on various prediction performance measures in the semi-supervised (SS) setting. In step I, the labeled data is used to obtain a non-parametrically calibrated estimate of the conditional risk function. In step II, SS estimates of the prediction accuracy parameters are constructed based on the estimated conditional risk function and the unlabeled data. We demonstrate that under mild regularity conditions the proposed estimators are consistent and asymptotically normal. Importantly, the asymptotic variance of the SS estimators is always smaller than that of the supervised counterparts under correct model specification. We also correct for potential overfitting bias in the SS estimators in finite sample with cross-validation and develop a perturbation resampling procedure to approximate their distributions. Our proposals are evaluated through extensive simulation studies and illustrated with two real EMR studies aiming to develop phenotyping algorithms for rheumatoid arthritis and multiple sclerosis.