mlsanityData to 14 Oct 2026

Effective Pre-Training Objectives for Transformer-based Autoencoders

arXiv 2210.13536 ↗First posted 24 Oct 2022cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we study trade-offs between efficiency, cost and accuracy when pre-training Transformer encoders with different pre-training objectives. For this purpose, we analyze features of common objectives and combine them to create new effective pre-training approaches. Specifically, we designed light token generators based on a straightforward statistical approach, which can replace ELECTRA computationally heavy generators, thus highly reducing cost. Our experiments also show that (i) there are more efficient alternatives to BERT's MLM, and (ii) it is possible to efficiently pre-train Transformer-based models using lighter generators without a significant drop in performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

22 of 36 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
10 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  3. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  4. 2020Efficient Transformers: A Surveycited by 790
  5. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  6. 2021How to Train BERT with an Academic Budgetcited by 80
  7. 2019TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selectioncited by 74
  8. 2020Multi-Stage Pre-training for Low-Resource Domain Adaptationcited by 24
  9. 2022Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selectioncited by 9
  10. 2019SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoderscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 0; 2026: 1.

3 papers.

  1. 2023Dynamic Masking Rate Schedules for MLM Pretrainingcited by 12
  2. 2023Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?cited by 2
  3. 2026UltraBERT: Beyond the Limit of BERT Pre-Trainingcited by 0