mlsanityData to 14 Oct 2026

nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources

arXiv 2309.02373 ↗First posted 5 Sep 2023cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

State-of-the-art language models like T5 have revolutionized the NLP landscape, but their computational demands hinder a large portion of the research community. To address this challenge, we present nanoT5, a specially-optimized PyTorch framework for efficient pre-training and fine-tuning of T5 models. Drawing on insights from optimizer differences and prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a single GPU in just 16 hours, without any loss in performance. With the introduction of this open-source framework, we hope to widen the accessibility to language modelling research and cater to the community's demand for more user-friendly T5 (Encoder-Decoder) implementations. We make our contributions, including configurations, codebase, pre-training insights, and pre-trained models, available to the public.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 5 Sep 2023.

Builds on

20 of 31 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2017Mixed Precision Trainingcited by 1,156
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
8 more
  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  2. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  3. 2022UL2: Unifying Language Learning Paradigmscited by 295
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  5. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  6. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  7. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  8. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6.

7 papers.

  1. 2024YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentationcited by 13
  2. 2024100K or 100 Days: Trade-offs when Pre-Training with Academic Resourcescited by 6
  3. 2023Sequence-to-Sequence Spanish Pre-trained Language Modelscited by 2
  4. 2024Exploring Chinese Humor Generation: A Study on Two-Part Allegorical Sayingscited by 1
  5. 2024Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformerscited by 1
  6. 2024When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantizationcited by 1
  7. 2024DEPTH: Discourse Education through Pre-Training Hierarchicallycited by 0