Calibrated Uncertainty Sampling for Active Learning
Not on the map. Why?
Abstract (arXiv)
We study the problem of actively learning a classifier with a low calibration error. One of the most popular Acquisition Functions (AFs) in pool-based Active Learning (AL) is querying by the model's uncertainty. However, we recognize that an uncalibrated uncertainty model on the unlabeled pool may significantly affect the AF effectiveness, leading to sub-optimal generalization and high calibration error on unseen data. Deep Neural Networks (DNNs) make it even worse as the model uncertainty from DNN is usually uncalibrated. Therefore, we propose a new AF by estimating calibration errors and query samples with the highest calibration error before leveraging DNN uncertainty. Specifically, we utilize a kernel calibration error estimator under the covariate shift and formally show that AL with this AF eventually leads to a bounded calibration error on the unlabeled pool and unseen test data. Empirically, our proposed method surpasses other AF baselines by having a lower calibration and generalization error across pool-based AL settings.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 3 Oct 2025.
Builds on
31 of 56 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
- 2017Deep Bayesian Active Learning with Image Datacited by 812
- 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
- 2018Do Deep Generative Models Know What They Don't Know?cited by 443
- 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
- 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
- 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
- 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
19 more
- 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
- 2021Batch Active Learning at Scalecited by 127
- 2019Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Costcited by 114
- 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
- 2019Distribution Calibration for Regressioncited by 88
- 2020Task-Aware Variational Adversarial Active Learningcited by 84
- 2017Deep Active Learning over the Long Tailcited by 82
- 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
- 2022Conformal Prediction Under Feedback Covariate Shift for Biomolecular Designcited by 53
- 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 44
- 2021Reducing Label Effort: Self-Supervised meets Active Learningcited by 42
- 2024Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)cited by 26
- 2023Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessmentcited by 25
- 2022Adaptive Temperature Scaling for Robust Calibration of Deep Neural Networkscited by 24
- 2024Transductive Active Learning: Theory and Applicationscited by 22
- 2023Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shiftscited by 12
- 2024A Cross-Domain Benchmark for Active Learningcited by 11
- 2022Active Learning and Novel Model Calibration Measurements for Automated Visual Inspection in Manufacturingcited by 4
- 2024CALICO: Confident Active Learning with Integrated Calibrationcited by 1
Built on by
Not cited yet.