mlsanityData to 14 Oct 2026

Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

arXiv 2605.20314 ↗First posted 19 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This work investigates the ``small-vs-large gap'', where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 33 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

29 of 33 references.

References · 29

  1. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  2. 2023Scaling Data-Constrained Language Modelscited by 403
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  4. 2020Query-Key Normalization for Transformerscited by 280
  5. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  6. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  7. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  8. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  9. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  10. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  11. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  12. 2023A Spectral Condition for Feature Learningcited by 109
17 more
  1. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  2. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  3. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  4. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 52
  5. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  6. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  7. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  8. 2021SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochscited by 22
  9. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  10. 2025Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regressioncited by 16
  11. 2024Emergent properties with repeated examplescited by 11
  12. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 11
  13. 2025Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributionscited by 8
  14. 2025Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularizationcited by 8
  15. 2026Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuningcited by 8
  16. 2026Full-Batch Gradient Descent Outperforms One-Pass SGD: Sample Complexity Separation in Single-Index Learningcited by 3
  17. 2021Why Does Multi-Epoch Training Help?cited by 1

Built on by

Not cited yet.