mlsanityData to 14 Oct 2026

Variance Reduction in SGD by Distributed Importance Sampling

arXiv 1511.06481 ↗First posted 20 Nov 2015stat.ML cs.LGcited by 79

Not on the map. Why?

Abstract (arXiv)

Humans are able to accelerate their learning by selecting training materials that are the most informative and at the appropriate level of difficulty. We propose a framework for distributing deep learning in which one set of workers search for the most informative examples in parallel while a single worker updates the model on examples selected by importance sampling. This leads the model to update using an unbiased estimate of the gradient which also has minimum variance when the sampling proposal is proportional to the L2-norm of the gradient. We show experimentally that this method reduces gradient variance even in a context where the cost of synchronization across machines cannot be ignored, and where the factors for importance sampling are not updated instantly across the training set.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 17; 2023: 14; 2024: 14; 2025: 12; 2026: 8.

79 papers.

Sort:

By area:

  1. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  2. 2021Efficient training of physics-informed neural networks via importance samplingcited by 110
  3. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  4. 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
  5. 2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
  6. 2023Identifying Spurious Biases Early in Training through the Lens of Simplicity Biascited by 44
  7. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  8. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 43
  9. 2022Neural Approaches to Conversational Information Retrievalcited by 42
  10. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  11. 2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
  12. 2022DELTA: Diverse Client Sampling for Fasting Federated Learningcited by 24
38 more
  1. 2022Mitigating Dataset Bias by Using Per-sample Gradientcited by 22
  2. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 20
  3. 2022Walk for Learning: A Random Walk Approach for Federated Learning from Heterogeneous Datacited by 19
  4. 2024EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Trainingcited by 18
  5. 2024Can Language Models Act as Knowledge Bases at Scale?cited by 17
  6. 2022Characterizing & Finding Good Data Orderings for Fast Convergence of Sequential Gradient Methodscited by 16
  7. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  8. 2022Federated Few-Shot Learning for Mobile NLPcited by 15
  9. 2021Variance Reduced Training with Stratified Sampling for Forecasting Modelscited by 14
  10. 2021Large Batch Experience Replaycited by 14
  11. 2024Diversified Batch Selection for Training Accelerationcited by 14
  12. 2024Unsupervised Concept Discovery Mitigates Spurious Correlationscited by 11
  13. 2021Disentangling Sampling and Labeling Bias for Learning in Large-Output Spacescited by 9
  14. 2024Adaptive Heterogeneous Client Sampling for Federated Learning over Wireless Networkscited by 9
  15. 2021Meta Learning on a Sequence of Imbalanced Domains with Difficulty Awarenesscited by 8
  16. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
  17. 2021Stochastic Reweighted Gradient Descentcited by 7
  18. 2023GaitMPL: Gait Recognition with Memory-Augmented Progressive Learningcited by 7
  19. 2023PA&DA: Jointly Sampling PAth and DAta for Consistent NAScited by 6
  20. 2021How Important is Importance Sampling for Deep Budgeted Training?cited by 5
  21. 2022Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learningcited by 5
  22. 2022A Survey of Dataset Refinement for Problems in Computer Vision Datasetscited by 5
  23. 2021One Backward from Ten Forward, Subsampling for Large-Scale Deep Learningcited by 4
  24. 2021Dynamic Gradient Aggregation for Federated Domain Adaptationcited by 4
  25. 2024From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modelingcited by 4
  26. 2025GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Modelscited by 4
  27. 2021Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learningcited by 3
  28. 2022Adaptively Weighted Data Augmentation Consistency Regularization for Robust Optimization under Concept Shiftcited by 3
  29. 2023AdaSelection: Accelerating Deep Learning Training through Data Subsamplingcited by 3
  30. 2024Efficient Backpropagation with Variance-Controlled Adaptive Samplingcited by 3
  31. 2024Efficient NeRF Optimization -- Not All Samples Remain Equally Hardcited by 3
  32. 2025Multiscale Training of Convolutional Neural Networkscited by 3
  33. 2025Differential-informed Sample Selection Accelerates Multimodal Contrastive Learningcited by 3
  34. 2025Variance Matters: Improving Domain Adaptation via Stratified Samplingcited by 3
  35. 2024Understanding the Training Speedup from Sampling with Approximate Lossescited by 2
  36. 2025Computational Budget Should Be Considered in Data Selectioncited by 2
  37. 2026Order Matters: Improving Domain Adaptation by Reordering Datacited by 2
  38. 2021Drill the Cork of Information Bottleneck by Inputting the Most Important Datacited by 1

50 most cited of 79 All 79 (JSON)