mlsanityData to 14 Oct 2026

Transfer training from smaller language model

arXiv 2104.11390 ↗First posted 23 Apr 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models have led to state-of-the-art accuracies across a range of tasks. However,training large language model needs massive computing resource, as more and more open source pre-training models are available, it is worthy to study how to take full advantage of available model. We find a method to save training time and resource cost by changing the small well-trained model to large model. We initialize a larger target model from a smaller source model by copy weight values from source model and padding with zeros or small initialization values on it to make the source and target model have approximate outputs, which is valid due to block matrix multiplication and residual connection in transformer structure. We test the target model on several data sets and find it is still comparable with the source model. When we continue training the target model, the training loss can start from a smaller value.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 25 references that are in the map. First posted on arXiv 23 Apr 2021.

Builds on

23 of 25 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2020Rethinking Attention with Performerscited by 1,601
11 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  2. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  4. 2020How Context Affects Language Models' Factual Predictionscited by 181
  5. 2018Improving the Transformer Translation Model with Document-Level Contextcited by 74
  6. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 70
  7. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  8. 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
  9. 2020Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Ordercited by 28
  10. 2020Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialogcited by 20
  11. 2021Efficient pre-training objectives for Transformerscited by 8

Built on by

Not cited yet.