# Gaussian Switch Sampling: A Second Order Approach to Active Learning

arXiv: https://arxiv.org/abs/2302.12018 · HTML: https://mlsanity.com/paper/2302.12018/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 64 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference](https://mlsanity.com/paper/1506.02158/) (2015)
- [Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima](https://mlsanity.com/paper/2111.01549/) (2021)
- [Discriminative Active Learning](https://mlsanity.com/paper/1907.06347/) (2019)
- [Deep Active Learning over the Long Tail](https://mlsanity.com/paper/1711.00941/) (2017)
- [SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios](https://mlsanity.com/paper/2107.00717/) (2021)
- [On Statistical Bias In Active Learning: How and When To Fix It](https://mlsanity.com/paper/2101.11665/) (2021)
- [Scalable Active Learning for Object Detection](https://mlsanity.com/paper/2004.04699/) (2020)
- [Gradients as a Measure of Uncertainty in Neural Networks](https://mlsanity.com/paper/2008.08030/) (2020)
- [Backpropagated Gradient Representations for Anomaly Detection](https://mlsanity.com/paper/2007.09507/) (2020)
- [CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognition](https://mlsanity.com/paper/1712.02463/) (2017)
- [Optimal Subsampling for Large Sample Logistic Regression](https://mlsanity.com/paper/1702.01166/) (2017)
- [Contrastive Explanations in Neural Networks](https://mlsanity.com/paper/2008.00178/) (2020)
- [Introspective Learning : A Two-Stage Approach for Inference in Neural Networks](https://mlsanity.com/paper/2209.08425/) (2022)
- [Novelty Detection Through Model-Based Characterization of Neural Networks](https://mlsanity.com/paper/2008.06094/) (2020)
- [Distorted Representation Space Characterization Through Backpropagated Gradients](https://mlsanity.com/paper/1908.09998/) (2019)
- [Contrastive Reasoning in Neural Networks](https://mlsanity.com/paper/2103.12329/) (2021)
- [On Warm-Starting Neural Network Training](https://mlsanity.com/paper/1910.08475/) (2019)

## Built on by (most cited first, 10 of 10)

- [FOCAL: A Cost-Aware Video Dataset for Active Learning](https://mlsanity.com/paper/2311.10591/) (2023)
- [Intelligent Multi-View Test Time Augmentation](https://mlsanity.com/paper/2406.08593/) (2024)
- [Clinical Trial Active Learning](https://mlsanity.com/paper/2307.11209/) (2023)
- [VOICE: Variance of Induced Contrastive Explanations to quantify Uncertainty in Neural Network Interpretability](https://mlsanity.com/paper/2406.00573/) (2024)
- [Counterfactual Gradients-based Quantification of Prediction Trust in Neural Networks](https://mlsanity.com/paper/2405.13758/) (2024)
- [Targeting Negative Flips in Active Learning using Validation Sets](https://mlsanity.com/paper/2411.10896/) (2024)
- [BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation](https://mlsanity.com/paper/2603.10828/) (2026)
- [Effective Data Selection for Seismic Interpretation through Disagreement](https://mlsanity.com/paper/2406.05149/) (2024)
- [Physics- and data-driven Active Learning of neural network representations for free energy functions of materials from statistical mechanics](https://mlsanity.com/paper/2503.07619/) (2025)
- [Are Candidate Models Really Needed for Active Learning?](https://mlsanity.com/paper/2605.14689/) (2026)

## Abstract (arXiv)

In active learning, acquisition functions define informativeness directly on the representation position within the model manifold. However, for most machine learning models (in particular neural networks) this representation is not fixed due to the training pool fluctuations in between active learning rounds. Therefore, several popular strategies are sensitive to experiment parameters (e.g. architecture) and do not consider model robustness to out-of-distribution settings. To alleviate this issue, we propose a grounded second-order definition of information content and sample importance within the context of active learning. Specifically, we define importance by how often a neural network "forgets" a sample during training - artifacts of second order representation shifts. We show that our definition produces highly accurate importance scores even when the model representations are constrained by the lack of training data. Motivated by our analysis, we develop Gaussian Switch Sampling (GauSS). We show that GauSS is setup agnostic and robust to anomalous distributions with exhaustive experiments on three in-distribution benchmarks, three out-of-distribution benchmarks, and three different architectures. We report an improvement of up to 5% when compared against four popular query strategies.