KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training
Not on the map. Why?
Abstract (arXiv)
This paper proposes a method for hiding the least-important samples during the training of deep neural networks to increase efficiency, i.e., to reduce the cost of training. Using information about the loss and prediction confidence during training, we adaptively find samples to exclude in a given epoch based on their contribution to the overall learning process, without significantly degrading accuracy. We explore the converge properties when accounting for the reduction in the number of SGD updates. Empirical results on various large-scale datasets and models used directly in image classification and segmentation show that while the with-replacement importance sampling algorithm performs poorly on large datasets, our method can reduce total training time by up to 22% impacting accuracy only by 0.4% compared to the baseline. Code available at https://github.com/TruongThaoNguyen/kakurenbo
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 16 Oct 2023.
Builds on
34 of 45 references.
References · 34
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2022Generative Adversarial Networkscited by 5,603
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2016Wide Residual Networkscited by 3,057
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2021Scaling Vision Transformerscited by 993
- 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
- 2019Green AIcited by 620
- 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
- 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
22 more
- 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
- 2021ResNet strikes back: An improved training procedure in timmcited by 365
- 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
- 2017Sampling Matters in Deep Embedding Learningcited by 285
- 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
- 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
- 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
- 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
- 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
- 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
- 2014Stochastic Optimization with Importance Samplingcited by 91
- 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
- 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
- 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
- 2019E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savingscited by 36
- 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
- 2018Exascale Deep Learning for Climate Analyticscited by 28
- 2020Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruningcited by 24
- 2015Even Faster Accelerated Coordinate Descent Using Non-Uniform Samplingcited by 23
- 2017Convergence Analysis of Distributed Stochastic Gradient Descent with Shufflingcited by 23
- 2022Pre-training Vision Transformers with Formula-driven Supervised Learningcited by 21
- 2021Clairvoyant Prefetching for Distributed Machine Learning I/Ocited by 14
Built on by
Citing papers per year: 2026: 1.
1 paper.