# One Epoch Is All You Need

arXiv: https://arxiv.org/abs/1906.06669 · HTML: https://mlsanity.com/paper/1906.06669/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2019.

## Built on by (most cited first, 30 of 41)

- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt](https://mlsanity.com/paper/2206.07137/) (2022)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Measuring Forgetting of Memorized Training Examples](https://mlsanity.com/paper/2207.00099/) (2022)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://mlsanity.com/paper/2212.14034/) (2022)
- [Limitations of Neural Collapse for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2202.08384/) (2022)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [ZipLM: Inference-Aware Structured Pruning of Language Models](https://mlsanity.com/paper/2302.04089/) (2023)
- [No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models](https://mlsanity.com/paper/2307.06440/) (2023)
- [MarIA: Spanish Language Models](https://mlsanity.com/paper/2107.07253/) (2021)
- [Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models](https://mlsanity.com/paper/2402.10430/) (2024)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework](https://mlsanity.com/paper/2109.05897/) (2021)
- [The Memory Perturbation Equation: Understanding Model's Sensitivity to Data](https://mlsanity.com/paper/2310.19273/) (2023)
- [Training Compute-Optimal Protein Language Models](https://mlsanity.com/paper/2411.02142/) (2024)
- [AdaPT: Fast Emulation of Approximate DNN Accelerators in PyTorch](https://mlsanity.com/paper/2203.04071/) (2022)
- [Emergent properties with repeated examples](https://mlsanity.com/paper/2410.07041/) (2024)
- [Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning](https://mlsanity.com/paper/2306.08590/) (2023)
- [SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks](https://mlsanity.com/paper/2506.10424/) (2025)
- [Sequence to sequence pretraining for a less-resourced Slovenian language](https://mlsanity.com/paper/2207.13988/) (2022)
- [Lessons learned from the evaluation of Spanish Language Models](https://mlsanity.com/paper/2212.08390/) (2022)
- [GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs](https://mlsanity.com/paper/2408.15300/) (2024)
- [Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions](https://mlsanity.com/paper/2401.04305/) (2024)
- [Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models](https://mlsanity.com/paper/2410.13088/) (2024)
- [Explanatory Argument Extraction of Correct Answers in Resident Medical Exams](https://mlsanity.com/paper/2312.00567/) (2023)
- [Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics](https://mlsanity.com/paper/2601.21698/) (2026)
- [Exposing Bias in Online Communities through Large-Scale Language Models](https://mlsanity.com/paper/2306.02294/) (2023)
- [Internal Data Repetition Destroys Language Models](https://mlsanity.com/paper/2606.24998/) (2026)

## Abstract (arXiv)

In unsupervised learning, collecting more data is not always a costly process unlike the training. For example, it is not hard to enlarge the 40GB WebText used for training GPT-2 by modifying its sampling methodology considering how many webpages there are in the Internet. On the other hand, given that training on this dataset already costs tens of thousands of dollars, training on a larger dataset naively is not cost-wise feasible. In this paper, we suggest to train on a larger dataset for only one epoch unlike the current practice, in which the unsupervised models are trained for from tens to hundreds of epochs. Furthermore, we suggest to adjust the model size and the number of iterations to be performed appropriately. We show that the performance of Transformer language model becomes dramatically improved in this way, especially if the original number of epochs is greater. For example, by replacing the training for 10 epochs with the one epoch training, this translates to 1.9-3.3x speedup in wall-clock time in our settings and more if the original number of epochs is greater. Under one epoch training, no overfitting occurs, and regularization method does nothing but slows down the training. Also, the curve of test loss over iterations follows power-law extensively. We compare the wall-clock time of the training of models with different parameter budget under one epoch training, and we show that size/iteration adjustment based on our proposed heuristics leads to 1-2.7x speedup in our cases. With the two methods combined, we achieve 3.3-5.1x speedup. Finally, we speculate various implications of one epoch training and size/iteration adjustment. In particular, based on our analysis we believe that we can reduce the cost to train the state-of-the-art models as BERT and GPT-2 dramatically, maybe even by the factor of 10.