Targeting Negative Flips in Active Learning using Validation Sets
Not on the map. Why?
Abstract (arXiv)
The performance of active learning algorithms can be improved in two ways. The often used and intuitive way is by reducing the overall error rate within the test set. The second way is to ensure that correct predictions are not forgotten when the training set is increased in between rounds. The former is measured by the accuracy of the model and the latter is captured in negative flips between rounds. Negative flips are samples that are correctly predicted when trained with the previous/smaller dataset and incorrectly predicted after additional samples are labeled. In this paper, we discuss improving the performance of active learning algorithms both in terms of prediction accuracy and negative flips. The first observation we make in this paper is that negative flips and overall error rates are decoupled and reducing one does not necessarily imply that the other is reduced. Our observation is important as current active learning algorithms do not consider negative flips directly and implicitly assume the opposite. The second observation is that performing targeted active learning on subsets of the unlabeled pool has a significant impact on the behavior of the active learning algorithm and influences both negative flips and prediction accuracy. We then develop ROSE - a plug-in algorithm that utilizes a small labeled validation set to restrict arbitrary active learning acquisition functions to negative flips within the unlabeled pool. We show that integrating a validation set results in a significant performance boost in terms of accuracy, negative flip rate reduction, or both.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 16 Nov 2024.
Builds on
34 of 58 references.
References · 34
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2016Learning without Forgettingcited by 2,465
- 2020Energy-based Out-of-distribution Detectioncited by 1,296
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
- 2017Deep Bayesian Active Learning with Image Datacited by 812
- 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
- 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
- 2015Towards Open Set Deep Networkscited by 533
22 more
- 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
- 2022Patches Are All You Need?cited by 201
- 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
- 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
- 2019Discriminative Active Learningcited by 97
- 2021SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarioscited by 83
- 2017Deep Active Learning over the Long Tailcited by 82
- 2020Scalable Active Learning for Object Detectioncited by 58
- 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 55
- 2020Backpropagated Gradient Representations for Anomaly Detectioncited by 54
- 2017CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognitioncited by 32
- 2021Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principlecited by 14
- 2023Gaussian Switch Sampling: A Second Order Approach to Active Learningcited by 10
- 2022Patient Aware Active Learning for Fine-Grained OCT Classificationcited by 9
- 2023FOCAL: A Cost-Aware Video Dataset for Active Learningcited by 9
- 2023Forgetful Active Learning with Switch Events: Efficient Sampling for Out-of-Distribution Datacited by 8
- 2022Open-Set Recognition with Gradient-Based Representationscited by 7
- 2022DECAL: DEployable Clinical Active Learningcited by 7
- 2023Explaining Deep Models through Forgettable Learning Dynamicscited by 7
- 2023Example Forgetting: A Novel Approach to Explain and Interpret Deep Neural Networks in Seismic Interpretationcited by 4
- 2024Transitional Uncertainty with Layered Intermediate Predictionscited by 4
- 2023Clinical Trial Active Learningcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.