mlsanityData to 7 Oct 2026

A Framework and Benchmark for Deep Batch Active Learning for Regression

arXiv 2203.09410 ↗First posted 17 Mar 2022stat.ML cs.LG cs.NEcited by 47

Not on the map. Why?

Abstract (arXiv)

The acquisition of labels for supervised learning can be expensive. To improve the sample efficiency of neural network regression, we study active learning methods that adaptively select batches of unlabeled data for labeling. We present a framework for constructing such methods out of (network-dependent) base kernels, kernel transformations, and selection methods. Our framework encompasses many existing Bayesian methods based on Gaussian process approximations of neural networks as well as non-Bayesian methods. Additionally, we propose to replace the commonly used last-layer features with sketched finite-width neural tangent kernels and to combine them with a novel clustering method. To evaluate different methods, we introduce an open-source benchmark consisting of 15 large tabular regression data sets. Our proposed method outperforms the state-of-the-art on our benchmark, scales to large data sets, and works out-of-the-box without adjusting the network architecture or training code. We provide open-source code that includes efficient implementations of all kernels, kernel transformations, and selection methods, and can be used for reproducing our results.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 59 of its 136 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

59 of 136 references.

References · 59

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  8. 2017Deep Bayesian Active Learning with Image Datacited by 812
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  11. 2018On Lazy Training in Differentiable Programmingcited by 671
  12. 2017Deep Neural Networks as Gaussian Processescited by 617
38 more
  1. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  2. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  3. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  4. 2020A Survey of Deep Active Learningcited by 547
  5. 2014OpenML: networked science in machine learningcited by 523
  6. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  7. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  8. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  9. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  10. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  11. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  12. 2014Sketching as a Tool for Numerical Linear Algebracited by 285
  13. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  14. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  15. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  16. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  17. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  18. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  19. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  20. 2019Diverse mini-batch Active Learningcited by 88
  21. 2021Gone Fishing: Neural Active Learning with Fisher Embeddingscited by 88
  22. 2017Deep Active Learning over the Long Tailcited by 82
  23. 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 72
  24. 2019Bayesian Batch Active Learning as Sparse Subset Approximationcited by 68
  25. 2021Simulation Intelligence: Towards a New Generation of Scientific Methodscited by 68
  26. 2019Batch Active Learning Using Determinantal Point Processescited by 42
  27. 2018Pool-Based Sequential Active Learning for Regressioncited by 33
  28. 2021Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networkscited by 32
  29. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  30. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  31. 2019Benchmarking the Neural Linear Model for Regressioncited by 31
  32. 2021Data Shapley Valuation for Efficient Batch Active Learningcited by 27
  33. 2021Properties of the After Kernelcited by 27
  34. 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
  35. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  36. 2018Dropout-based Active Learning for Regressioncited by 24
  37. 2021Neural Active Learning with Performance Guaranteescited by 22
  38. 2018Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensemblescited by 20

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 10; 2024: 10; 2025: 11; 2026: 13.

47 papers.

Sort:

By area:

  1. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  2. 2022Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluationscited by 31
  3. 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
  4. 2023Low-rank extended Kalman filtering for online learning of neural networks from streaming datacited by 23
  5. 2024Active Learning for Neural PDE Solverscited by 22
  6. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 21
  7. 2023Uncertainty-biased molecular dynamics for learning uniformly accurate interatomic potentialscited by 16
  8. 2023KECOR: Kernel Coding Rate Maximization for Active 3D Object Detectioncited by 13
  9. 2023Black-Box Batch Active Learning for Regressioncited by 12
  10. 2022Transfer learning for chemically accurate interatomic neural network potentialscited by 11
  11. 2024Active Fine-Tuning of Multi-Task Policiescited by 8
  12. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
35 more
  1. 2024A prediction rigidity formalism for low-cost uncertainties in trained neural networkscited by 7
  2. 2024lpNTK: Better Generalisation with Less Data via Sample Interaction During Learningcited by 6
  3. 2025Generalization Bounds and Stopping Rules for Learning with Self-Selected Datacited by 5
  4. 2024Learning from the Best: Active Learning for Wireless Communicationscited by 4
  5. 2025Cohort-Based Active Modality Acquisitioncited by 4
  6. 2025Active Learning with Selective Time-Step Acquisition for PDEscited by 4
  7. 2026Uncertainty Quantification for Flow-Based Vision-Language-Action Modelscited by 4
  8. 2024Progressive Generalization Risk Reduction for Data-Efficient Causal Effect Estimationcited by 3
  9. 2025Enhancing Treatment Effect Estimation via Active Learning: A Counterfactual Covering Perspectivecited by 3
  10. 2026Pretrained Model Representations as Acquisition Signals for Active Learning of MLIPscited by 3
  11. 2023BatchGFN: Generative Flow Networks for Batch Active Learningcited by 2
  12. 2024Active Few-Shot Fine-Tuningcited by 2
  13. 2026Force-Aware Neural Tangent Kernels for Scalable and Robust Active Learning of MLIPscited by 2
  14. 2023Does Deep Active Learning Work in the Wild?cited by 1
  15. 2023Efficient Epistemic Uncertainty Estimation in Regression Ensemble Models Using Pairwise-Distance Estimatorscited by 1
  16. 2024Gradient Aligned Regression via Pairwise Lossescited by 1
  17. 2025No Free Lunch in Active Learning: LLM Embedding Quality Dictates Query Strategy Successcited by 1
  18. 2025Pointwise confidence estimation in the non-linear ℓ2-regularized least squarescited by 1
  19. 2025MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysiscited by 1
  20. 2025ActiveCQ: Active Estimation of Causal Quantitiescited by 1
  21. 2026Observationally Informed Adaptive Causal Experimental Designcited by 1
  22. 2026Learning Where to Simulate: Generative Active Sampling for Online PDE Surrogate Trainingcited by 1
  23. 2023Model Uncertainty based Active Learning on Tabular Data using Boosted Treescited by 0
  24. 2024regAL: Python Package for Active Learning of Regression Problemscited by 0
  25. 2025Efficient Data Selection for Training Genomic Perturbation Modelscited by 0
  26. 2025ActivePusher: Active Learning and Planning with Residual Physics for Nonprehensile Manipulationcited by 0
  27. 2025Adaptive Data Selection for Multi-Layer Perceptron Training: A Sub-linear Value-Driven Methodcited by 0
  28. 2026Active Transfer Bagging: A New Approach for Accelerated Active Learning Acquisition of Data by Combined Transfer Learning and Bagging Based Modelscited by 0
  29. 2026CAAL: Confidence-Aware Active Learning for Heteroscedastic Atmospheric Regressioncited by 0
  30. 2026Feature Weighting Improves Pool-Based Sequential Active Learning for Regressioncited by 0
  31. 2026A Mutual Information Lower Bound for Multimodal Regression Active Learningcited by 0
  32. 2026Data-Efficient Neural Operator Training via Physics-Based Active Learningcited by 0
  33. 2026How Low Can You Go? Active Learning for Sparse Model Discovery in the Ultra-Low-Data Limitcited by 0
  34. 2026Distillation of Foundation Models for Time-dependent PDEscited by 0
  35. 2026AdaptNTK: Adaptive Uncertainty Quantification and Active Learning for Neural Network Potentialscited by 0