mlsanityData to 14 Oct 2026

Cramming: Training a Language Model on a Single GPU in One Day

arXiv 2212.14034 ↗First posted 28 Dec 2022cs.CL cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment where training language models is out of reach for most researchers and practitioners. While most in the community are asking how to push the limits of extreme computation, we ask the opposite question: How far can we get with a single GPU in just one day? We investigate the downstream performance achievable with a transformer-based language model trained completely from scratch with masked language modeling for a single day on a single consumer GPU. Aside from re-analyzing nearly all components of the pretraining pipeline for this scenario and providing a modified pipeline with performance close to BERT, we investigate why scaling down is hard, and which modifications actually improve performance in this scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. Through the lens of scaling laws, we categorize a range of recent improvements to training and architecture and discuss their merit and practical applicability (or lack thereof) for the limited compute setting.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 107 of its 146 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

107 of 146 references.

References · 107

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,413
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  11. 2019Root Mean Square Layer Normalizationcited by 1,480
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  13. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  14. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  15. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  16. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  17. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  18. 2017Mixed Precision Trainingcited by 1,156
  19. 2019When Does Label Smoothing Help?cited by 988
  20. 2018Neural Network Acceptability Judgmentscited by 958
  21. 2020On Layer Normalization in the Transformer Architecturecited by 922
  22. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  23. 2021Membership Inference Attacks From First Principlescited by 878
  24. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  25. 2020Efficient Transformers: A Surveycited by 790
  26. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  27. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  28. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  29. 2016Language Modeling with Gated Convolutional Networkscited by 702
  30. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  31. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  32. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  33. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  34. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  35. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  36. 2021Explaining Neural Scaling Lawscited by 363
  37. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  38. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341

50 most cited of 107 All 107 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 25; 2024: 27; 2025: 19; 2026: 10.

81 papers.

Sort:

By area:

  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  2. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
  3. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 104
  4. 2023Data Distillation: A Surveycited by 87
  5. 2025REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjectscited by 80
  6. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  7. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  8. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
  9. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  10. 2023Simplifying Transformer Blockscited by 37
  11. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 35
  12. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 34
38 more
  1. 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 28
  2. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  3. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  4. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 22
  5. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  6. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  7. 2023CLIMB: Curriculum Learning for Infant-inspired Model Buildingcited by 20
  8. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 19
  9. 2024Encryption-Friendly LLM Architecturecited by 17
  10. 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
  11. 2024Diffusion Guided Language Modelingcited by 15
  12. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 12
  13. 2023The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Computecited by 10
  14. 2024FaBERT: Pre-training BERT on Persian Blogscited by 10
  15. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  16. 2024PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Softwarecited by 9
  17. 2024You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNetcited by 9
  18. 2023Data-Efficient French Language Modeling with CamemBERTacited by 8
  19. 2023nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resourcescited by 7
  20. 2024Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretrainingcited by 7
  21. 2025Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curationcited by 7
  22. 2023V1T: large-scale mouse V1 response prediction using a Vision Transformercited by 6
  23. 2024HDT: Hierarchical Document Transformercited by 6
  24. 2024100K or 100 Days: Trade-offs when Pre-Training with Academic Resourcescited by 6
  25. 2024Symmetric Dot-Product Attention for Efficient Training of BERT Language Modelscited by 5
  26. 2025Leaner Transformers: More Heads, Less Depthcited by 5
  27. 2023Navigating Scaling Laws: Compute Optimality in Adaptive Model Trainingcited by 4
  28. 2023Exponentially Faster Language Modellingcited by 4
  29. 2024Sustainable self-supervised learning for speech representationscited by 4
  30. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  31. 2023Adapting to the Low-Resource Double-Bind: Investigating Low-Compute Methods on Low-Resource African Languagescited by 3
  32. 2023Approximating CKY with Transformerscited by 3
  33. 2023Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Buildingcited by 3
  34. 2024Efficient Backpropagation with Variance-Controlled Adaptive Samplingcited by 3
  35. 2024N-gram Prediction and Word Difference Representations for Language Modelingcited by 3
  36. 2024ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Modelscited by 3
  37. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  38. 2025Scaling Laws for Data-Efficient Visual Transfer Learningcited by 3

50 most cited of 81 All 81 (JSON)