mlsanityData to 14 Oct 2026

Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples

arXiv 2406.03944 ↗First posted 6 Jun 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Neural Network-based active learning (NAL) is a cost-effective data selection technique that utilizes neural networks to select and train on a small subset of samples. While existing work successfully develops various effective or theory-justified NAL algorithms, the understanding of the two commonly used query criteria of NAL: uncertainty-based and diversity-based, remains in its infancy. In this work, we try to move one step forward by offering a unified explanation for the success of both query criteria-based NAL from a feature learning view. Specifically, we consider a feature-noise data model comprising easy-to-learn or hard-to-learn features disrupted by noise, and conduct analysis over 2-layer NN-based NALs in the pool-based scenario. We provably show that both uncertainty-based and diversity-based NAL are inherently amenable to one and the same principle, i.e., striving to prioritize samples that contain yet-to-be-learned features. We further prove that this shared principle is the key to their success-achieve small test error within a small labeled set. Contrastingly, the strategy-free passive learning exhibits a large test error due to the inadequate learning of yet-to-be-learned features, necessitating resort to a significantly larger label complexity for a sufficient test error reduction. Experimental results validate our findings.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 54 of its 104 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

54 of 104 references.

References · 54

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017Deep Bayesian Active Learning with Image Datacited by 819
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  5. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  6. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  7. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  8. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  9. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
  10. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  11. 2019Discriminative Active Learningcited by 98
  12. 2019Deep Active Learning: Unified and Principled Method for Query and Trainingcited by 93
38 more
  1. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  2. 2019Diverse mini-batch Active Learningcited by 89
  3. 2022A Comparative Survey of Deep Active Learningcited by 89
  4. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  5. 2023White-Box Transformers via Sparse Rate Reductioncited by 80
  6. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  7. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  8. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  9. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  10. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 55
  11. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  12. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  13. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  14. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  15. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 46
  16. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  17. 2017Active Learning for Convolutional Neural Networks: A Core-Set Approachcited by 36
  18. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  19. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  20. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  21. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 28
  22. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  23. 2022Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernelscited by 25
  24. 2021Neural Active Learning with Performance Guaranteescited by 22
  25. 2020On the Neural Tangent Kernel of Deep Networks with Orthogonal Initializationcited by 21
  26. 2023The Benefits of Mixup for Feature Learningcited by 19
  27. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  28. 2021On the Equivalence between Neural Network and Support Vector Machinecited by 17
  29. 2021Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspectivecited by 14
  30. 2022Provably Learning Diverse Features in Multi-View Data with Midpoint Mixupcited by 14
  31. 2022Active Learning with Neural Networks: Insights from Nonparametric Statisticscited by 13
  32. 2023Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptronscited by 12
  33. 2024Improved Particle Approximation Error for Mean Field Neural Networkscited by 12
  34. 2021Benign Overfitting in Adversarially Robust Linear Classificationcited by 11
  35. 2023Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problemscited by 11
  36. 2023Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Datacited by 10
  37. 2023Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescited by 8
  38. 2021A Neural Pre-Conditioning Active Learning Algorithm to Reduce Label Complexitycited by 7

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 3.

5 papers.

  1. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  2. 2026DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Discrete Diffusion Modelscited by 3
  3. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 2
  4. 2024MelissaDL x Breed: Towards Data-Efficient On-line Supervised Training of Multi-parametric Surrogates with Active Learningcited by 1
  5. 2026Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guaranteescited by 0