mlsanityData to 7 Oct 2026

Importance Sampling for Minibatches

arXiv 1602.02283 ↗First posted 6 Feb 2016cs.LG math.OC stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

Minibatching is a very well studied and highly popular technique in supervised learning, used by practitioners due to its ability to accelerate training through better utilization of parallel processing power and reduction of stochastic variance. Another popular technique is importance sampling -- a strategy for preferential sampling of more important examples also capable of accelerating the training process. However, despite considerable effort by the community in these areas, and due to the inherent technical difficulty of the problem, there is no existing work combining the power of importance sampling with the strength of minibatching. In this paper we propose the first em importance sampling for minibatches and give simple and rigorous complexity analysis of its performance. We illustrate on synthetic problems that for training data of certain properties, our sampling can lead to several orders of magnitude improvement in training time. We then test the new sampling on several popular datasets, and show that the improvement can reach an order of magnitude.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 6; 2023: 4; 2024: 4; 2025: 9; 2026: 4.

36 papers.

Sort:

  1. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  2. 2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 66
  3. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  4. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  5. 2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
  6. 2021Reweighting Augmented Samples by Minimizing the Maximal Expected Losscited by 21
  7. 2022Amortized Variational Inference: A Systematic Reviewcited by 16
  8. 2021Variance Reduced Training with Stratified Sampling for Forecasting Modelscited by 13
  9. 2021Adaptive Importance Sampling meets Mirror Descent: a Bias-variance tradeoffcited by 13
  10. 2022DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Samplingcited by 13
  11. 2023Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?cited by 10
  12. 2021Adaptive Importance Sampling for Finite-Sum Optimization and Sampling with Decreasing Step-Sizescited by 9
24 more
  1. 2021A General Theory for Client Sampling in Federated Learningcited by 8
  2. 2021Stochastic Reweighted Gradient Descentcited by 7
  3. 2021Uniform Sampling over Episode Difficultycited by 6
  4. 2021Adaptive Client Sampling in Federated Learning via Online Learning with Bandit Feedbackcited by 6
  5. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  6. 2024The Entrapment Problem in Random Walk Decentralized Learningcited by 4
  7. 2024Parallel Momentum Methods Under Biased Gradient Estimationscited by 2
  8. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  9. 2022Modelling calibration uncertainty in networks of environmental sensorscited by 1
  10. 2026Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gapscited by 1
  11. 2022Tricks and Plugins to GBM on Images and Sequencescited by 0
  12. 2023A Negative Result on Gradient Matching for Selective Backpropcited by 0
  13. 2024Unsupervised Federated Optimization at the Edge: D2D-Enabled Learning without Labelscited by 0
  14. 2024Grad Queue : A probabilistic framework to reinforce sparse gradientscited by 0
  15. 2025Multi-Timescale Primal Dual Hybrid Gradient with Application to Distributed Optimizationcited by 0
  16. 2025Efficient Training of Deep Networks using Guided Spectral Data Selection: A Step Toward Learning What You Needcited by 0
  17. 2025Tricks and Plug-ins for Gradient Boosting in Image Classificationcited by 0
  18. 2025Tricks and Plug-ins for Gradient Boosting with Transformerscited by 0
  19. 2025HyperCore: Coreset Selection under Noise via Hypersphere Modelscited by 0
  20. 2025SubZeroCore: A Submodular Approach with Zero Training for Coreset Selectioncited by 0
  21. 2025The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weightscited by 0
  22. 2026Decentralized Learning via Random Walk with Jumpscited by 0
  23. 2026Disagreement-Regularized Importance Sampling for Adversarial Label Corruptioncited by 0
  24. 2026Bayesian Tensor Autoencoder with Physics-informed Predictive Prior for Multi-dimensional Time Series Anomaly Detectioncited by 0