# Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples

arXiv: https://arxiv.org/abs/1704.07433 · HTML: https://mlsanity.com/paper/1704.07433/

## Key facts

- Cited by 147 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2017.

## Built on by (most cited first, 30 of 147)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)
- [Coverage-centric Coreset Selection for High Pruning Rates](https://mlsanity.com/paper/2210.15809/) (2022)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [Robust Training under Label Noise by Over-parameterization](https://mlsanity.com/paper/2202.14026/) (2022)
- [Jo-SRC: A Contrastive Approach for Combating Noisy Labels](https://mlsanity.com/paper/2103.13029/) (2021)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [Sample Selection with Uncertainty of Losses for Learning with Noisy Labels](https://mlsanity.com/paper/2106.00445/) (2021)
- [UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning](https://mlsanity.com/paper/2203.14542/) (2022)
- [Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference](https://mlsanity.com/paper/2110.10031/) (2021)
- [Large-scale Dataset Pruning with Dynamic Uncertainty](https://mlsanity.com/paper/2306.05175/) (2023)
- [Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering](https://mlsanity.com/paper/2107.02331/) (2021)
- [Part-based Pseudo Label Refinement for Unsupervised Person Re-identification](https://mlsanity.com/paper/2203.14675/) (2022)
- [Self-Damaging Contrastive Learning](https://mlsanity.com/paper/2106.02990/) (2021)
- [Sample Selection for Fair and Robust Training](https://mlsanity.com/paper/2110.14222/) (2021)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)
- [Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification](https://mlsanity.com/paper/2109.05427/) (2021)
- [On Tilted Losses in Machine Learning: Theory and Applications](https://mlsanity.com/paper/2109.06141/) (2021)
- [Data Valuation Without Training of a Model](https://mlsanity.com/paper/2301.00930/) (2023)
- [Learning from Noisy Labels for Entity-Centric Information Extraction](https://mlsanity.com/paper/2104.08656/) (2021)
- [Instance-specific and Model-adaptive Supervision for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2211.11335/) (2022)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding](https://mlsanity.com/paper/2312.05328/) (2023)
- [Meta-learning with an Adaptive Task Scheduler](https://mlsanity.com/paper/2110.14057/) (2021)
- [Towards Open Set Video Anomaly Detection](https://mlsanity.com/paper/2208.11113/) (2022)
- [Making Your First Choice: To Address Cold Start Problem in Vision Active Learning](https://mlsanity.com/paper/2210.02442/) (2022)
- [Procrustean Training for Imbalanced Deep Learning](https://mlsanity.com/paper/2104.01769/) (2021)
- [CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning](https://mlsanity.com/paper/2202.05613/) (2022)

## Abstract (arXiv)

Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of mini-batch SGD, and the proximity of the correct class probability to the decision threshold. Extensive experimental results on six datasets show that our methods reliably improve accuracy in various network architectures, including additional gains on top of other popular training techniques, such as residual learning, momentum, ADAM, batch normalization, dropout, and distillation.