mlsanityData to 14 Oct 2026

Data-Efficient Training by Evolved Sampling

arXiv 2509.23461 ↗First posted 27 Sep 2025cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Data selection is designed to accelerate learning with preserved performance. To achieve this, a fundamental thought is to identify informative data samples with significant contributions to the training. In this work, we propose Evolved Sampling (ES), a simple yet effective framework for dynamic sampling along the training process. This method conducts em batch em level data selection based on the dynamics of losses and augmented loss differences, which enables flexible frequency tuning, and hence significantly reduces the back propagation time with maintained model performance. Due to its conciseness, ES is also readily extensible to incorporate em set em level data selection (to form ES with pruning, ESWP) for further accelerations. As a plug-and-play framework, ES(WP) consistently achieves lossless training accelerations across various pre-training and post-training tasks, saving up to nearly 45% wall-clock time. Our results motivate further investigations on the data efficiency aspect of modern large-scale machine learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 55 of its 67 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

55 of 67 references.

References · 55

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  7. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2015Prioritized Experience Replaycited by 1,099
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
38 more
  1. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  2. 2017Large Batch Training of Convolutional Networkscited by 535
  3. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  5. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  6. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  7. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  8. 2021Dataset Condensation with Distribution Matchingcited by 365
  9. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  10. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  11. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  12. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  13. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  14. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  15. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
  16. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  17. 2018Adversarial Active Learning for Deep Networks: a Margin Based Approachcited by 158
  18. 2021Active Learning by Acquiring Contrastive Examplescited by 156
  19. 2019Distributionally Robust Language Modelingcited by 143
  20. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 133
  21. 2015Online Batch Selection for Faster Training of Neural Networkscited by 122
  22. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  23. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  24. 2024How to Train Data-Efficient LLMscited by 93
  25. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  26. 2015Variance Reduction in SGD by Distributed Importance Samplingcited by 79
  27. 2018On Coresets for Logistic Regressioncited by 67
  28. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51
  29. 2021Accelerating Deep Learning with Dynamic Data Pruningcited by 43
  30. 2024Data Selection via Optimal Control for Language Modelscited by 38
  31. 2024Capturing the Temporal Dependence of Training Data Influencecited by 35
  32. 2017Biased Importance Sampling for Deep Neural Network Trainingcited by 31
  33. 2016Minimizing the Maximal Loss: How and Why?cited by 30
  34. 2023Towards a statistical theory of data selection under weak supervisioncited by 27
  35. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  36. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 20
  37. 2023Towards Accelerated Model Training via Bayesian Data Selectioncited by 11
  38. 2021Stochastic Reweighted Gradient Descentcited by 7

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.