mlsanityData to 7 Oct 2026

METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

arXiv 2204.06644 ↗First posted 13 Apr 2022cs.LG cs.AI cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training strategy has demonstrated sample-efficiency to pretrain models at the scale of hundreds of millions of parameters. In this work, we conduct a comprehensive empirical study, and propose a recipe, namely "Model generated dEnoising TRaining Objective" (METRO), which incorporates some of the best modeling techniques developed recently to speed up, stabilize, and enhance pretrained language models without compromising model effectiveness. The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks. More importantly, METRO-LM are efficient in that they often outperform previous large models with significantly smaller model sizes and lower pretraining cost.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 55 of its 73 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

55 of 73 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
38 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  13. 2020On Layer Normalization in the Transformer Architecturecited by 906
  14. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  15. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  16. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  17. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  18. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  19. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  20. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  21. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  22. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  23. 2017Automated Curriculum Learning for Neural Networkscited by 251
  24. 2018A Simple Method for Commonsense Reasoningcited by 223
  25. 2020Understanding the Difficulty of Training Transformerscited by 210
  26. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  27. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  28. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  29. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  30. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  31. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  32. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  33. 2020Adversarial Training for Large Neural Language Modelscited by 109
  34. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  35. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  36. 2021How to Train BERT with an Academic Budgetcited by 80
  37. 2022Neural Approaches to Conversational Information Retrievalcited by 41
  38. 2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 9; 2024: 4; 2025: 1.

20 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  3. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  4. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  5. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  6. 2022Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarizationcited by 29
  7. 2022Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUEcited by 24
  8. 2022Beyond English-Centric Bitexts for Better Multilingual Language Representation Learningcited by 16
  9. 2023Sparse Backpropagation for MoE Trainingcited by 16
  10. 2022Improving Short Text Classification With Augmented Data Using GPT-3cited by 11
  11. 2023AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processingcited by 7
  12. 2024ReALM: Reference Resolution As Language Modelingcited by 7
8 more
  1. 2022What is Wrong with Language Models that Can Not Tell a Story?cited by 4
  2. 2023Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformerscited by 1
  3. 2023Fast-ELECTRA for Efficient Pre-trainingcited by 1
  4. 2023Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Wayscited by 1
  5. 2023Labels Need Prompts Too: Mask Matching for Natural Language Understanding Taskscited by 1
  6. 2024Vygotsky Distance: Measure for Benchmark Task Similaritycited by 1
  7. 2025Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related taskscited by 1
  8. 2024SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detectioncited by 0