mlsanityData to 14 Oct 2026

Towards Accelerated Model Training via Bayesian Data Selection

arXiv 2308.10544 ↗First posted 21 Aug 2023cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Mislabeled, duplicated, or biased data in real-world scenarios can lead to prolonged training and even hinder model convergence. Traditional solutions prioritizing easy or hard samples lack the flexibility to handle such a variety simultaneously. Recent work has proposed a more reasonable data selection principle by examining the data's impact on the model's generalization loss. However, its practical adoption relies on less principled approximations and additional holdout data. This work solves these problems by leveraging a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models. The resulting algorithm is efficient and easy to implement. We perform extensive empirical studies on challenging benchmarks with considerable data noise and imbalance in the online batch selection scenario, and observe superior training efficiency over competitive baselines. Notably, on the challenging WebVision benchmark, our method can achieve similar predictive performance with significantly fewer training iterations than leading data selection methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

35 of 47 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023Segment Anythingcited by 8,234
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  10. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  11. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  12. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
23 more
  1. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  2. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  3. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  4. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  5. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  6. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  7. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  8. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  9. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  10. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  11. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  12. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  13. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  14. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  15. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  16. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  17. 2015Online Batch Selection for Faster Training of Neural Networkscited by 122
  18. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  19. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  20. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  21. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  22. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51
  23. 2020BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 3.

11 papers.

  1. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  2. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 15
  3. 2024Diversified Batch Selection for Training Accelerationcited by 14
  4. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  5. 2025OASIS: Online Sample Selection for Continual Visual Instruction Tuningcited by 9
  6. 2024Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classificationcited by 6
  7. 2025Partial Forward Blocking: A Novel Data Pruning Paradigm for Lossless Training Accelerationcited by 3
  8. 2026GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selectioncited by 2
  9. 2025Data-Efficient Training by Evolved Samplingcited by 0
  10. 2026SLAP: Stratified Loss-based Pruning for On-Policy Data-Efficient Instruction Tuningcited by 0
  11. 2026Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuningcited by 0