FoundationLayerNorm: Scaling BERT and GPT to 1,000 Layers
Not on the map. Why?
Abstract (arXiv)
The mainstream BERT/GPT model contains only 10 to 20 layers, and there is little literature to discuss the training of deep BERT/GPT. This paper proposes a simple yet effective method to stabilize BERT and GPT training. We successfully scale up BERT and GPT to 1,000 layers, which is an order of magnitude deeper than previous BERT and GPT. The proposed method FoundationLayerNormalization enables efficient training of deep neural networks and is validated at the 1000-layer scale.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 14 references that are in the map. First posted on arXiv 9 Apr 2022.
Builds on
11 of 14 references.
References · 11
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
- 2020Understanding the Difficulty of Training Transformerscited by 210
- 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
- 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
- 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
Built on by
Not cited yet.