# Targeting Negative Flips in Active Learning using Validation Sets

arXiv: https://arxiv.org/abs/2411.10896 · HTML: https://mlsanity.com/paper/2411.10896/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 58 references that are in the map.
- First posted on arXiv 16 Nov 2024.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Patches Are All You Need?](https://mlsanity.com/paper/2201.09792/) (2022)
- [Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima](https://mlsanity.com/paper/2111.01549/) (2021)
- [Discriminative Active Learning](https://mlsanity.com/paper/1907.06347/) (2019)
- [Deep Active Learning over the Long Tail](https://mlsanity.com/paper/1711.00941/) (2017)
- [SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios](https://mlsanity.com/paper/2107.00717/) (2021)
- [Scalable Active Learning for Object Detection](https://mlsanity.com/paper/2004.04699/) (2020)
- [Positive-Congruent Training: Towards Regression-Free Model Updates](https://mlsanity.com/paper/2011.09161/) (2020)
- [Backpropagated Gradient Representations for Anomaly Detection](https://mlsanity.com/paper/2007.09507/) (2020)
- [CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognition](https://mlsanity.com/paper/1712.02463/) (2017)
- [Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle](https://mlsanity.com/paper/2105.14559/) (2021)
- [Gaussian Switch Sampling: A Second Order Approach to Active Learning](https://mlsanity.com/paper/2302.12018/) (2023)
- [Patient Aware Active Learning for Fine-Grained OCT Classification](https://mlsanity.com/paper/2206.11485/) (2022)
- [FOCAL: A Cost-Aware Video Dataset for Active Learning](https://mlsanity.com/paper/2311.10591/) (2023)
- [Forgetful Active Learning with Switch Events: Efficient Sampling for Out-of-Distribution Data](https://mlsanity.com/paper/2301.05106/) (2023)
- [Open-Set Recognition with Gradient-Based Representations](https://mlsanity.com/paper/2206.08229/) (2022)
- [DECAL: DEployable Clinical Active Learning](https://mlsanity.com/paper/2206.10120/) (2022)

## Built on by (most cited first, 1 of 1)

- [BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation](https://mlsanity.com/paper/2603.10828/) (2026)

## Abstract (arXiv)

The performance of active learning algorithms can be improved in two ways. The often used and intuitive way is by reducing the overall error rate within the test set. The second way is to ensure that correct predictions are not forgotten when the training set is increased in between rounds. The former is measured by the accuracy of the model and the latter is captured in negative flips between rounds. Negative flips are samples that are correctly predicted when trained with the previous/smaller dataset and incorrectly predicted after additional samples are labeled. In this paper, we discuss improving the performance of active learning algorithms both in terms of prediction accuracy and negative flips. The first observation we make in this paper is that negative flips and overall error rates are decoupled and reducing one does not necessarily imply that the other is reduced. Our observation is important as current active learning algorithms do not consider negative flips directly and implicitly assume the opposite. The second observation is that performing targeted active learning on subsets of the unlabeled pool has a significant impact on the behavior of the active learning algorithm and influences both negative flips and prediction accuracy. We then develop ROSE - a plug-in algorithm that utilizes a small labeled validation set to restrict arbitrary active learning acquisition functions to negative flips within the unlabeled pool. We show that integrating a validation set results in a significant performance boost in terms of accuracy, negative flip rate reduction, or both.