mlsanityData to 14 Oct 2026

Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels

arXiv 2206.12569 ↗First posted 25 Jun 2022stat.ML cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

We propose a new method for approximating active learning acquisition strategies that are based on retraining with hypothetically-labeled candidate data points. Although this is usually infeasible with deep networks, we use the neural tangent kernel to approximate the result of retraining, and prove that this approximation works asymptotically even in an active learning setup -- approximating "look-ahead" selection criteria with far less computation required. This also enables us to conduct sequential active learning, i.e. updating the model in a streaming regime, without needing to retrain the model with SGD after adding each new data point. Moreover, our querying strategy, which better understands how the model's predictions will change by adding new data points in comparison to the standard ("myopic") criteria, beats other look-ahead strategies by large margins, and achieves equal or better performance compared to state-of-the-art methods on several benchmark datasets in pool-based active learning.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 25 Jun 2022.

Builds on

27 of 51 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2016Wide Residual Networkscited by 3,070
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2017Deep Bayesian Active Learning with Image Datacited by 819
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  8. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  11. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  12. 2019Learning Loss for Active Learningcited by 336
15 more
  1. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  2. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  3. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  4. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
  5. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  6. 2017FALKON: An Optimal Large Scale Kernel Methodcited by 84
  7. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  8. 2019Bayesian Generative Active Deep Learningcited by 57
  9. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  10. 2019Batch Active Learning Using Determinantal Point Processescited by 43
  11. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  12. 2019Epistemic Uncertainty Samplingcited by 24
  13. 2016Active and Continuous Exploration with Deep Neural Networks and Expected Model Output Changescited by 18
  14. 2020Semi-supervised Batch Active Learning via Bilevel Optimizationcited by 16
  15. 2021Meta-Learning with Neural Tangent Kernelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 10; 2025: 4; 2026: 2.

25 papers.

Sort:

By area:

  1. 2024Learning Dynamics of LLM Finetuningcited by 95
  2. 2024A Survey on Deep Active Learning: Recent Advances and New Frontierscited by 54
  3. 2024PINNACLE: PINN Adaptive ColLocation and Experimental points selectioncited by 32
  4. 2023Algorithm Selection for Deep Active Learning with Imbalanced Datasetscited by 18
  5. 2024Generalized Coverage for More Robust Low-Budget Active Learningcited by 16
  6. 2023Differentially Private Neural Tangent Kernels for Privacy-Preserving Data Generationcited by 15
  7. 2024Uncertainty Herding: One Active Learning Method for All Label Budgetscited by 15
  8. 2023KECOR: Kernel Coding Rate Maximization for Active 3D Object Detectioncited by 13
  9. 2023Improved Algorithm for Deep Active Learning under Imbalance via Optimal Separationcited by 8
  10. 2024Querying Easily Flip-flopped Samples for Deep Active Learningcited by 7
  11. 2024AHA: Human-Assisted Out-of-Distribution Generalization and Detectioncited by 7
  12. 2024Out-of-Distribution Learning with Human Feedbackcited by 6
13 more
  1. 2023Training-Free Neural Active Learning with Initialization-Robustness Guaranteescited by 5
  2. 2023Re-weighting Tokens: A Simple and Effective Active Learning Strategy for Named Entity Recognitioncited by 5
  3. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5
  4. 2024Deep Active Learning in the Open Worldcited by 5
  5. 2023Synaptic Weight Distributions Depend on the Geometry of Plasticitycited by 4
  6. 2023NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coveragecited by 3
  7. 2026Learning Dynamics in RL Post-Training for Language Modelscited by 2
  8. 2023Exploring Active Learning in Meta-Learning: Enhancing Context Set Labelingcited by 1
  9. 2025Improving Task Diversity in Label Efficient Supervised Finetuning of LLMscited by 1
  10. 2025Accumulative SGD Influence Estimation for Data Attributioncited by 1
  11. 2026The Speed-up Factor: A Quantitative Multi-Iteration Active Learning Performance Metriccited by 1
  12. 2025ActivePusher: Active Learning and Planning with Residual Physics for Nonprehensile Manipulationcited by 0
  13. 2025Diffusion-Driven Two-Stage Active Learning for Low-Budget Semantic Segmentationcited by 0