mlsanityData to 14 Oct 2026

Training a T5 Using Lab-sized Resources

arXiv 2208.12097 ↗First posted 25 Aug 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Training large neural language models on large datasets is resource- and time-intensive. These requirements create a barrier to entry, where those with fewer resources cannot build competitive models. This paper presents various techniques for making it possible to (a) train a large language model using resources that a modest research lab might have, and (b) train it in a reasonable amount of time. We provide concrete recommendations for practitioners, which we illustrate with a case study: a T5 model for Danish, the first for this language.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 25 Aug 2022.

Builds on

23 of 28 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  8. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  10. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  12. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
11 more
  1. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  2. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  3. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  4. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  5. 2017Towards Accurate Binary Convolutional Neural Networkcited by 149
  6. 2020The Cost of Training NLP Models: A Concise Overviewcited by 98
  7. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 80
  8. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  9. 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
  10. 2020Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMTcited by 18
  11. 2020The Danish Gigaword Projectcited by 17

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2023Robot-Enabled Construction Assembly with Automated Sequence Planning based on ChatGPT: RoboGPTcited by 11
  2. 2023Danish Foundation Modelscited by 4
  3. 2023Vocabulary-level Memory Efficiency for Language Model Fine-tuningcited by 2
  4. 2024ChatGPT Alternative Solutions: Large Language Models Surveycited by 2
  5. 2024SnakModel: Lessons Learned from Training an Open Danish Large Language Modelcited by 0