mlsanityData to 14 Oct 2026

Soft Random Sampling: A Theoretical and Empirical Analysis

arXiv 2311.12727 ↗First posted 21 Nov 2023cs.LG cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Soft random sampling (SRS) is a simple yet effective approach for efficient training of large-scale deep neural networks when dealing with massive data. SRS selects a subset uniformly at random with replacement from the full data set in each epoch. In this paper, we conduct a theoretical and empirical analysis of SRS. First, we analyze its sampling dynamics including data coverage and occupancy. Next, we investigate its convergence with non-convex objective functions and give the convergence rate. Finally, we provide its generalization performance. We empirically evaluate SRS for image recognition on CIFAR10 and automatic speech recognition on Librispeech and an in-house payload dataset to demonstrate its effectiveness. Compared to existing coreset-based data selection methods, SRS offers a better accuracy-efficiency trade-off. Especially on real-world industrial scale data sets, it is shown to be a powerful training strategy with significant speedup and competitive performance with almost no additional computing cost.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 52 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

21 of 52 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  7. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  8. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  9. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  10. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  11. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
  12. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 147
9 more
  1. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64
  2. 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
  3. 2017English Conversational Telephone Speech Recognition by Humans and Machinescited by 40
  4. 2020Coresets for Robust Training of Neural Networks against Noisy Labelscited by 24
  5. 2021Training Data Subset Selection for Regression with Controlled Generalization Errorcited by 14
  6. 2021SVP-CF: Selection via Proxy for Collaborative Filtering Datacited by 14
  7. 2021Data Summarization via Bilevel Optimizationcited by 8
  8. 2019Realizing Petabyte Scale Acoustic Modelingcited by 6
  9. 2021Bayesian subset selection and variable importance for interpretable prediction and classificationcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024NEBULA: Neural Empirical Bayes Under Latent Representations for Efficient and Controllable Design of Molecular Librariescited by 3
  2. 2025Do we need equivariant models for molecule generation?cited by 2