# Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning

arXiv: https://arxiv.org/abs/2305.18424 · HTML: https://mlsanity.com/paper/2305.18424/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 79 references that are in the map.
- First posted on arXiv 28 May 2023.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt](https://mlsanity.com/paper/2206.07137/) (2022)
- [GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning](https://mlsanity.com/paper/2012.10630/) (2020)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Dataset Condensation via Efficient Synthetic-Data Parameterization](https://mlsanity.com/paper/2205.14959/) (2022)
- [Adversarial Active Learning for Deep Networks: a Margin Based Approach](https://mlsanity.com/paper/1802.09841/) (2018)
- [DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning](https://mlsanity.com/paper/2204.08499/) (2022)
- [Contextual Diversity for Active Learning](https://mlsanity.com/paper/2008.05723/) (2020)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)

## Built on by (most cited first, 26 of 26)

- [DsDm: Model-Aware Dataset Selection with Datamodels](https://mlsanity.com/paper/2401.12926/) (2024)
- [Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning](https://mlsanity.com/paper/2311.13613/) (2023)
- [Training-Free Dataset Pruning for Instance Segmentation](https://mlsanity.com/paper/2503.00828/) (2025)
- [Modyn: Data-Centric Machine Learning Pipeline Orchestration](https://mlsanity.com/paper/2312.06254/) (2023)
- [General Information Metrics for Improving AI Model Training Efficiency](https://mlsanity.com/paper/2501.02004/) (2025)
- [Disentangling the Roles of Representation and Selection in Data Pruning](https://mlsanity.com/paper/2507.03648/) (2025)
- [Looking Beyond the Known: Towards a Data Discovery Guided Open-World Object Detection](https://mlsanity.com/paper/2510.00303/) (2025)
- [SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Sampling](https://mlsanity.com/paper/2311.15134/) (2023)
- [Gini Coefficient as a Unified Metric for Evaluating Many-versus-Many Similarity in Vector Spaces](https://mlsanity.com/paper/2411.07983/) (2024)
- [Effective Data Pruning through Score Extrapolation](https://mlsanity.com/paper/2506.09010/) (2025)
- [Rethinking Representativeness and Diversity in Dynamic Data Selection](https://mlsanity.com/paper/2603.04981/) (2026)
- [OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework](https://mlsanity.com/paper/2606.08574/) (2026)
- [Deep Submodular Peripteral Networks](https://mlsanity.com/paper/2403.08199/) (2024)
- [One Period to Rule Them All: Identifying Critical Learning Periods in Deep Networks](https://mlsanity.com/paper/2506.15954/) (2025)
- [Autoguided Online Data Curation for Diffusion Model Training](https://mlsanity.com/paper/2509.15267/) (2025)
- [SHaSaM: Submodular Hard Sample Mining for Fair Facial Attribute Recognition](https://mlsanity.com/paper/2602.05162/) (2026)
- [Rethinking Dataset Distillation: Hard Truths about Soft Labels](https://mlsanity.com/paper/2604.18811/) (2026)
- [Are Coreset Selection Methods Worth Their Cost?](https://mlsanity.com/paper/2609.22894/) (2026)
- [Hierarchical mutual distillation for multi-view fusion: Learning from all possible view combinations](https://mlsanity.com/paper/2411.10077/) (2024)
- [Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes](https://mlsanity.com/paper/2511.17399/) (2025)
- [Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment](https://mlsanity.com/paper/2604.07306/) (2026)
- [Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training](https://mlsanity.com/paper/2605.14773/) (2026)
- [Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling](https://mlsanity.com/paper/2605.23198/) (2026)
- [RCAP: Robust, Class-Aware, Probabilistic Dynamic Dataset Pruning](https://mlsanity.com/paper/2606.11761/) (2026)
- [Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration](https://mlsanity.com/paper/2606.12913/) (2026)
- [ERank in Latent Space as an Image-Complexity and Richness Measure](https://mlsanity.com/paper/2607.19315/) (2026)

## Abstract (arXiv)

Methods for carefully selecting or generating a small set of training data to learn from, i.e., data pruning, coreset selection, and data distillation, have been shown to be effective in reducing the ever-increasing cost of training neural networks. Behind this success are rigorously designed strategies for identifying informative training examples out of large datasets. However, these strategies come with additional computational costs associated with subset selection or data distillation before training begins, and furthermore, many are shown to even under-perform random sampling in high data compression regimes. As such, many data pruning, coreset selection, or distillation methods may not reduce 'time-to-accuracy', which has become a critical efficiency measure of training deep neural networks over large datasets. In this work, we revisit a powerful yet overlooked random sampling strategy to address these challenges and introduce an approach called Repeated Sampling of Random Subsets (RSRS or RS2), where we randomly sample the subset of training data for each epoch of model training. We test RS2 against thirty state-of-the-art data pruning and data distillation methods across four datasets including ImageNet. Our results demonstrate that RS2 significantly reduces time-to-accuracy compared to existing techniques. For example, when training on ImageNet in the high-compression regime (using less than 10% of the dataset each epoch), RS2 yields accuracy improvements up to 29% compared to competing pruning methods while offering a runtime reduction of 7x. Beyond the above meta-study, we provide a convergence analysis for RS2 and discuss its generalization capability. The primary goal of our work is to establish RS2 as a competitive baseline for future data selection or distillation techniques aimed at efficient training.