Margin-based sampling in high dimensions: When being active is less efficient than staying passive
Not on the map. Why?
Abstract (arXiv)
It is widely believed that given the same labeling budget, active learning (AL) algorithms like margin-based active learning achieve better predictive performance than passive learning (PL), albeit at a higher computational cost. Recent empirical evidence suggests that this added cost might be in vain, as margin-based AL can sometimes perform even worse than PL. While existing works offer different explanations in the low-dimensional regime, this paper shows that the underlying mechanism is entirely different in high dimensions: we prove for logistic regression that PL outperforms margin-based AL even for noiseless data and when using the Bayes optimal decision boundary for sampling. Insights from our proof indicate that this high-dimensional phenomenon is exacerbated when the separation between the classes is small. We corroborate this intuition with experiments on 20 high-dimensional datasets spanning a diverse range of applications, from finance and histology to chemistry and computer vision.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 50 references that are in the map. First posted on arXiv 1 Dec 2022.
Builds on
24 of 50 references.
References · 24
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
- 2018Robustness May Be at Odds with Accuracycited by 873
- 2017Deep Bayesian Active Learning with Image Datacited by 812
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2019Benign Overfitting in Linear Regressioncited by 544
- 2014OpenML: networked science in machine learningcited by 523
- 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
- 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
- 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
- 2018Adversarial Active Learning for Deep Networks: a Margin Based Approachcited by 155
- 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 131
12 more
- 2019Discriminative Active Learningcited by 97
- 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
- 2019Deep Active Learning: Unified and Principled Method for Query and Trainingcited by 92
- 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 72
- 2018Adversarial Sampling for Active Learningcited by 51
- 2020Precise Statistical Analysis of Classification Accuracies for Adversarial Trainingcited by 49
- 2016A Benchmark and Comparison of Active Learning for Logistic Regressioncited by 36
- 2015Convergence Rates of Active Learning for Maximum Likelihood Estimationcited by 25
- 2021Self-training Converts Weak Learners to Strong Learners in Mixture Modelscited by 19
- 2021Interpolation can hurt robust generalization even when there is no noisecited by 15
- 2018Efficient active learning of sparse halfspacescited by 13
- 2018On the Relationship between Data Efficiency and Error for Uncertainty Samplingcited by 12
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 2.
3 papers.