Fast-ELECTRA for Efficient Pre-training
Not on the map. Why?
Abstract (arXiv)
ELECTRA pre-trains language models by detecting tokens in a sequence that have been replaced by an auxiliary model. Although ELECTRA offers a significant boost in efficiency, its potential is constrained by the training cost brought by the auxiliary model. Notably, this model, which is jointly trained with the main model, only serves to assist the training of the main model and is discarded post-training. This results in a substantial amount of training cost being expended in vain. To mitigate this issue, we propose Fast-ELECTRA, which leverages an existing language model as the auxiliary model. To construct a learning curriculum for the main model, we smooth its output distribution via temperature scaling following a descending schedule. Our approach rivals the performance of state-of-the-art ELECTRA-style pre-training methods, while significantly eliminating the computation and memory cost brought by the joint training of the auxiliary model. Our method also reduces the sensitivity to hyper-parameters and enhances the pre-training stability.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 11 Oct 2023.
Builds on
29 of 35 references.
References · 29
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
17 more
- 2016Training Deep Nets with Sublinear Memory Costcited by 825
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
- 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
- 2020Understanding the Difficulty of Training Transformerscited by 210
- 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
- 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
- 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
- 2016Memory-Efficient Backpropagation Through Timecited by 78
- 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
- 2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28
- 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
- 2021Multi-head or Single-head? An Empirical Comparison for Transformer Trainingcited by 15
- 2020MC-BERT: Efficient Language Pre-Training via a Meta Controllercited by 14
- 2022Pretraining Text Encoders with Adversarial Mixture of Training Signal Generatorscited by 9
- 2021Learning to Sample Replacements for ELECTRA Pre-Trainingcited by 7
- 2022Language Model Pre-training on True Negativescited by 1
Built on by
Citing papers per year: 2024: 1.
1 paper.