mlsanityData to 7 Oct 2026

How to Train BERT with an Academic Budget

arXiv 2104.07705 ↗First posted 15 Apr 2021cs.CL cs.AI cs.LGcited by 80

Not on the map. Why?

Abstract (arXiv)

While large language models a la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well-funded industry labs can afford. How can one train such models with a more modest budget? We present a recipe for pretraining a masked language model in 24 hours using a single low-end deep learning server. We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT-base on GLUE tasks at a fraction of the original pretraining cost.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

29 of 42 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
17 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2019CamemBERT: a Tasty French Language Modelcited by 368
  8. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  9. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  10. 2018Scaling Neural Machine Translationcited by 204
  11. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  12. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  13. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
  14. 2020Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensemblescited by 43
  15. 2019Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraintscited by 36
  16. 2020Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedupcited by 31
  17. 2020Optimal Subarchitecture Extraction For BERTcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 19; 2023: 27; 2024: 16; 2025: 11; 2026: 2.

80 papers.

Sort:

By area:

  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  2. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  3. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  5. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  6. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  7. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  8. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  9. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  10. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  11. 2022Pretraining Without Attentioncited by 51
  12. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
38 more
  1. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  2. 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimizationcited by 43
  3. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 43
  4. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  5. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  6. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 33
  7. 2021Curriculum learning for language modelingcited by 32
  8. 2021Pre-train or Annotate? Domain Adaptation with a Constrained Budgetcited by 27
  9. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 26
  10. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  11. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
  12. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  13. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
  14. 2023CLIMB: Curriculum Learning for Infant-inspired Model Buildingcited by 20
  15. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20
  16. 2022DCT-Former: Efficient Self-Attention with Discrete Cosine Transformcited by 12
  17. 2021Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processingcited by 11
  18. 2022Incorporating Context into Subword Vocabulariescited by 11
  19. 2023Does compressing activations help model parallel training?cited by 11
  20. 2023The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deploymentcited by 11
  21. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  22. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  23. 2023How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectivescited by 9
  24. 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noisecited by 9
  25. 2023Data-Efficient French Language Modeling with CamemBERTacited by 8
  26. 2023Biomedical Language Models are Robust to Sub-optimal Tokenizationcited by 8
  27. 2022Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Tokencited by 7
  28. 2022ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERTcited by 7
  29. 2023Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Expertscited by 7
  30. 2024Don't Touch My Diacriticscited by 7
  31. 2024Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformerscited by 7
  32. 2022Training a Vision Transformer from scratch in less than 24 hours with 1 GPUcited by 6
  33. 2024Orchid: Flexible and Data-Dependent Convolution for Sequence Modelingcited by 6
  34. 2024100K or 100 Days: Trade-offs when Pre-Training with Academic Resourcescited by 6
  35. 2022Transformers with Learnable Activation Functionscited by 5
  36. 2023Surveying (Dis)Parities and Concerns of Compute Hungry NLP Researchcited by 5
  37. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  38. 2023Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrewcited by 5

50 most cited of 80 All 80 (JSON)