Importance Sampling for Stochastic Gradient Descent in Deep Neural Networks
Not on the map. Why?
Abstract (arXiv)
Stochastic gradient descent samples uniformly the training set to build an unbiased gradient estimate with a limited number of samples. However, at a given step of the training process, some data are more helpful than others to continue learning. Importance sampling for training deep neural networks has been widely studied to propose sampling schemes yielding better performance than the uniform sampling scheme. After recalling the theory of importance sampling for deep learning, this paper reviews the challenges inherent to this research area. In particular, we propose a metric allowing the assessment of the quality of a given sampling scheme; and we study the interplay between the sampling scheme and the optimizer used.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 29 Mar 2023.
Builds on
17 of 26 references.
References · 17
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2015Prioritized Experience Replaycited by 1,070
- 2017Sampling Matters in Deep Embedding Learningcited by 285
- 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
- 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
- 2014Stochastic Optimization with Importance Samplingcited by 91
- 2015Variance Reduction in SGD by Distributed Importance Samplingcited by 79
- 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
5 more
- 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
- 2017Learning What Data to Learncited by 21
- 2013On Optimal Probabilities in Stochastic Coordinate Descent Methodscited by 14
- 2021Large Batch Experience Replaycited by 14
- 2016Accelerating Deep Neural Network Training with Inconsistent Stochastic Gradient Descentcited by 6
Built on by
Citing papers per year: 2023: 1.
1 paper.
- 2023A Large Deviations Perspective on Policy Gradient Algorithms13 Nov 2023