mlsanityData to 14 Oct 2026

Training-Free Neural Active Learning with Initialization-Robustness Guarantees

arXiv 2306.04454 ↗First posted 7 Jun 2023cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Existing neural active learning algorithms have aimed to optimize the predictive performance of neural networks (NNs) by selecting data for labelling. However, other than a good predictive performance, being robust against random parameter initializations is also a crucial requirement in safety-critical applications. To this end, we introduce our expected variance with Gaussian processes (EV-GP) criterion for neural active learning, which is theoretically guaranteed to select data points which lead to trained NNs with both (a) good predictive performances and (b) initialization robustness. Importantly, our EV-GP criterion is training-free, i.e., it does not require any training of the NN during data selection, which makes it computationally efficient. We empirically demonstrate that our EV-GP criterion is highly correlated with both initialization robustness and generalization performance, and show that it consistently outperforms baseline methods in terms of both desiderata, especially in situations with limited initial data or large batch sizes.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 53 of its 90 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

53 of 90 references.

References · 53

  1. 2016Wide Residual Networkscited by 3,070
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  4. 2017Deep Bayesian Active Learning with Image Datacited by 819
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  6. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2020A Survey of Deep Active Learningcited by 554
  10. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  11. 2017On Kernelized Multi-armed Banditscited by 352
  12. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
38 more
  1. 2019Variational Adversarial Active Learningcited by 284
  2. 2020Neural Architecture Search without Trainingcited by 244
  3. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  4. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  5. 2014Lazier Than Lazy Greedycited by 99
  6. 2019Discriminative Active Learningcited by 98
  7. 2019Deep Active Learning: Unified and Principled Method for Query and Trainingcited by 93
  8. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  9. 2020Task-Aware Variational Adversarial Active Learningcited by 84
  10. 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
  11. 2020Active Domain Adaptation via Clustering Uncertainty-weighted Embeddingscited by 72
  12. 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 65
  13. 2021Optimal Order Simple Regret for Gaussian Process Banditscited by 54
  14. 2019Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpointcited by 42
  15. 2021Neural Contextual Bandits without Regretcited by 40
  16. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  17. 2021Differentially Private Federated Bayesian Optimization with Distributed Explorationcited by 32
  18. 2020Finding trainable sparse networks through Neural Tangent Transfercited by 28
  19. 2021NASI: Label- and Data-agnostic Neural Architecture Search at Initializationcited by 28
  20. 2022Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernelscited by 25
  21. 2022Sample-Then-Optimize Batch Neural Thompson Samplingcited by 25
  22. 2017Decentralized High-Dimensional Bayesian Optimization with Factor Graphscited by 23
  23. 2020Top-k Ranking Bayesian Optimizationcited by 22
  24. 2021Neural Active Learning with Performance Guaranteescited by 22
  25. 2021Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outlierscited by 22
  26. 2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21
  27. 2021Value-at-Risk Optimization with Gaussian Processescited by 20
  28. 2022Unifying and Boosting Gradient-Based Training-Free Neural Architecture Searchcited by 17
  29. 2022Federated Neural Banditscited by 17
  30. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimizationcited by 16
  31. 2015Gaussian Process Planning with Lipschitz Continuous Reward Functions: Towards Unifying Bayesian Optimization, Active Learning, and Beyondcited by 15
  32. 2019Bayesian Optimization with Binary Auxiliary Informationcited by 15
  33. 2020Private Outsourced Bayesian Optimizationcited by 15
  34. 2020An Information-Theoretic Framework for Unifying Active Learning Problemscited by 14
  35. 2020R2-B2: Recursive Reasoning-Based Bayesian Optimization for No-Regret Learning in Gamescited by 11
  36. 2015Near-Optimal Active Learning of Multi-Output Gaussian Processescited by 10
  37. 2022Bayesian Optimization under Stochastic Delayed Feedbackcited by 10
  38. 2020Nonmyopic Gaussian Process Optimization with Macro-Actionscited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024PINNACLE: PINN Adaptive ColLocation and Experimental points selectioncited by 32
  2. 2024Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictionscited by 14
  3. 2024Neural Active Learning Beyond Banditscited by 10
  4. 2025PIED: Physics-Informed Experimental Design for Inverse Problemscited by 2
  5. 2025Uncovering Scaling Laws for Large Language Models via Inverse Problemscited by 0