mlsanityData to 14 Oct 2026

Online Batch Selection for Faster Training of Neural Networks

arXiv 1511.06343 ↗First posted 19 Nov 2015cs.LG cs.NE math.OCcited by 122

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are commonly trained using stochastic non-convex optimization procedures, which are driven by gradient information estimated on fractions (batches) of the dataset. While it is commonly accepted that batch size is an important parameter for offline tuning, the benefits of online selection of batches remain poorly understood. We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam. As the loss function to be minimized for the whole dataset is an aggregation of loss functions of individual datapoints, intuitively, datapoints with the greatest loss should be considered (selected in a batch) more frequently. However, the limitations of this intuition and the proper control of the selection pressure over time are open questions. We propose a simple strategy where all datapoints are ranked w.r.t. their latest known loss value and the probability to be selected decays exponentially as a function of rank. Our experimental results on the MNIST dataset suggest that selecting batches speeds up both AdaDelta and Adam by a factor of about 5.

Key facts. Cited by 122 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 12; 2023: 25; 2024: 23; 2025: 21; 2026: 24.

122 papers.

Sort:

By area:

  1. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  2. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  3. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  4. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  5. 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
  6. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 56
  7. 2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
  8. 2021Instance-wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translationcited by 45
  9. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  10. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 43
  11. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  12. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
38 more
  1. 2024Data curation via joint example selection further accelerates multimodal learningcited by 33
  2. 2023Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understandingcited by 28
  3. 2025Learning to Reason at the Frontier of Learnabilitycited by 23
  4. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 21
  5. 2021A Tale Of Two Long Tailscited by 19
  6. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  7. 2024EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Trainingcited by 18
  8. 2021Churn Reduction via Distillationcited by 17
  9. 2023Irreducible Curriculum for Language Model Pretrainingcited by 16
  10. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  11. 2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
  12. 2024Does Negative Sampling Matter? A Review with Insights into its Theory and Applicationscited by 15
  13. 2024Nonparametric Teaching of Implicit Neural Representationscited by 15
  14. 2025Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Trainingcited by 15
  15. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 15
  16. 2021Variance Reduced Training with Stratified Sampling for Forecasting Modelscited by 14
  17. 2021Large Batch Experience Replaycited by 14
  18. 2023DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning with Static and Streaming Datasetcited by 14
  19. 2023Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networkscited by 14
  20. 2024Diversified Batch Selection for Training Accelerationcited by 14
  21. 2024Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalizationcited by 12
  22. 2023Towards Accelerated Model Training via Bayesian Data Selectioncited by 11
  23. 2023MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuningcited by 9
  24. 2025ELIP: Enhanced Visual-Language Foundation Models for Image Retrievalcited by 9
  25. 2021BulletTrain: Accelerating Robust Neural Network Training via Boundary Example Miningcited by 8
  26. 2023Mini-Batch Optimization of Contrastive Losscited by 8
  27. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
  28. 2024Active Generation for Image Classificationcited by 8
  29. 2021Locally Adaptive Label Smoothing for Predictive Churncited by 7
  30. 2021Stochastic Reweighted Gradient Descentcited by 7
  31. 2021Prioritized training on points that are learnable, worth learning, and not yet learned (workshop version)cited by 7
  32. 2022To Store or Not? Online Data Selection for Federated Learning with Limited Storagecited by 7
  33. 2023GaitMPL: Gait Recognition with Memory-Augmented Progressive Learningcited by 7
  34. 2024Evolution-aware VAriance (EVA) Coreset Selection for Medical Image Classificationcited by 7
  35. 2023Data Optimization in Deep Learning: A Surveycited by 6
  36. 2024Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classificationcited by 6
  37. 2025Towards Active Synthetic Data Generation for Finetuning Language Modelscited by 6
  38. 2021How Important is Importance Sampling for Deep Budgeted Training?cited by 5

50 most cited of 122 All 122 (JSON)