mlsanityData to 7 Oct 2026

Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt

arXiv 2206.07137 ↗First posted 14 Jun 2022cs.LG cs.AI cs.CLcited by 233

Not on the map. Why?

Abstract (arXiv)

Training on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training, we introduce Reducible Holdout Loss Selection (RHO-LOSS), a simple but principled technique which selects approximately those points for training that most reduce the model's generalization loss. As a result, RHO-LOSS mitigates the weaknesses of existing data selection methods: techniques from the optimization literature typically select 'hard' (e.g. high loss) points, but such points are often noisy (not learnable) or less task-relevant. Conversely, curriculum learning prioritizes 'easy' points, but such points need not be trained on once learned. In contrast, RHO-LOSS selects points that are learnable, worth learning, and not yet learnt. RHO-LOSS trains in far fewer steps than prior art, improves accuracy, and speeds up training on a wide range of datasets, hyperparameters, and architectures (MLPs, CNNs, and BERT). On the large web-scraped image dataset Clothing-1M, RHO-LOSS trains in 18x fewer steps and reaches 2% higher final accuracy than uniform data shuffling.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Builds on 40 of its 63 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

40 of 63 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
28 more
  1. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  3. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  4. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  5. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  6. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  7. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  8. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  9. 2018An Empirical Model of Large-Batch Trainingcited by 268
  10. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  11. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  12. 2019Probabilistic End-to-end Noise Correction for Learning with Noisy Labelscited by 210
  13. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  14. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  15. 2017Deep Learning is Robust to Massive Label Noisecited by 164
  16. 2019Practical Deep Learning with Bayesian Principlescited by 158
  17. 2018Large scale distributed neural network training through online distillationcited by 144
  18. 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
  19. 2019Image Classification with Deep Learning in the Presence of Noisy Labels: A Surveycited by 101
  20. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  21. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  22. 2015Variance Reduction in SGD by Distributed Importance Samplingcited by 79
  23. 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 72
  24. 2019Cold Case: The Lost MNIST Digitscited by 61
  25. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51
  26. 2019One Epoch Is All You Needcited by 41
  27. 2017Biased Importance Sampling for Deep Neural Network Trainingcited by 31
  28. 2019AutoAssist: A Framework to Accelerate Training of Deep Neural Networkscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 38; 2024: 58; 2025: 62; 2026: 62.

233 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  3. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  4. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  5. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  6. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  7. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  8. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  9. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  10. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  11. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  12. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
38 more
  1. 2024How to Train Data-Efficient LLMscited by 91
  2. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  3. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  4. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  5. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  6. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  7. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  8. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  9. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  10. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  11. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  12. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
  13. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  14. 2024Data Selection via Optimal Control for Language Modelscited by 38
  15. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  16. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 35
  17. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  18. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  19. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  20. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  21. 2025Self-Improving LLM Agents at Test-Timecited by 30
  22. 2022Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantitiescited by 28
  23. 2023Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understandingcited by 28
  24. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 28
  25. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  26. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  27. 2023Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learningcited by 26
  28. 2024Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Textscited by 26
  29. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  30. 2023Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Informationcited by 24
  31. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  32. 2024Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretrainingcited by 24
  33. 2023Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Leastcited by 22
  34. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 22
  35. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  36. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  37. 2022General Intelligence Requires Rethinking Explorationcited by 20
  38. 2024CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learningcited by 20

50 most cited of 233 All 233 (JSON)