# Label-Efficient Monitoring of Classification Models via Stratified Importance Sampling

arXiv: https://arxiv.org/abs/2601.22326 · HTML: https://mlsanity.com/paper/2601.22326/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 13 of 13)

- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [PMLB v1.0: An open source dataset collection for benchmarking machine learning methods](https://mlsanity.com/paper/2012.00058/) (2020)
- [Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation](https://mlsanity.com/paper/2202.06881/) (2022)
- [Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation](https://mlsanity.com/paper/2406.04291/) (2024)
- [A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation](https://mlsanity.com/paper/2406.07320/) (2024)
- [Sequential Harmful Shift Detection Without Labels](https://mlsanity.com/paper/2412.12910/) (2024)
- [Estimating Model Performance Under Covariate Shift Without Labels](https://mlsanity.com/paper/2401.08348/) (2024)
- [Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories](https://mlsanity.com/paper/2109.05720/) (2021)
- [In Search of an Entity Resolution OASIS: Optimal Asymptotic Sequential Importance Sampling](https://mlsanity.com/paper/1703.00617/) (2017)
- [Needle in a Haystack: Label-Efficient Evaluation under Extreme Class Imbalance](https://mlsanity.com/paper/2006.06963/) (2020)
- [Recursive Neyman Algorithm for Optimum Sample Allocation under Box Constraints on Sample Sizes in Strata](https://mlsanity.com/paper/2304.07034/) (2023)
- [Always Strengthen Your Strengths: A Drift-Aware Incremental Learning Framework for CTR Prediction](https://mlsanity.com/paper/2304.09062/) (2023)

## Abstract (arXiv)

Monitoring the performance of classification models in production is critical yet challenging due to strict labeling budgets, one-shot batch acquisition of labels and extremely low error rates. We propose a general framework based on Stratified Importance Sampling (SIS) that directly addresses these constraints in model monitoring. While SIS has previously been applied in specialized domains, our theoretical analysis establishes its broad applicability to the monitoring of classification models. Under mild conditions, SIS yields unbiased estimators with strict finite-sample mean squared error (MSE) improvements over both importance sampling (IS) and stratified random sampling (SRS). The framework does not rely on optimally defined proposal distributions or strata: even with noisy proxies and sub-optimal stratification, SIS can improve estimator efficiency compared to IS or SRS individually, though extreme proposal mismatch may limit these gains. Experiments across binary and multiclass tasks demonstrate consistent efficiency improvements under fixed label budgets, underscoring SIS as a principled, label-efficient, and operationally lightweight methodology for post-deployment model monitoring.