mlsanityData to 14 Oct 2026

Emergent properties with repeated examples

arXiv 2410.07041 ↗First posted 9 Oct 2024cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that two-set training - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

24 of 40 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2020In Search of Lost Domain Generalizationcited by 1,054
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  12. 2019Benign Overfitting in Linear Regressioncited by 549
12 more
  1. 2023Scaling Data-Constrained Language Modelscited by 403
  2. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  3. 2021Deep learning: a statistical viewpointcited by 240
  4. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  5. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  6. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  7. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  8. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  9. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  10. 2021Linear algebra with transformerscited by 63
  11. 2019One Epoch Is All You Needcited by 41
  12. 2023Learning the greatest common divisor: explaining transformer predictionscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 37
  2. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  3. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  4. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  5. 2026Efficient RL Training for LLMs with Experience Replaycited by 9
  6. 2026Scaling Domain Data Repetition in LLM Pretrainingcited by 2
  7. 2025Int2Int: a framework for mathematics with transformerscited by 1
  8. 2025Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretabilitycited by 0
  9. 2026Improving ML Attacks on LWE with Data Repetition and Stepwise Regressioncited by 0
  10. 2026Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biasescited by 0
  11. 2026Generating Special Triangulations with Transformerscited by 0