Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
Not on the map. Why?
Abstract (arXiv)
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5. We study various designs to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise. Key aspects under study include the decoding target, the location of the RTD head, and the masking pattern. Based on these studies, we develop a new model, METRO-T0, which is pretrained using the redesigned ELECTRA-Style pretraining strategies and then prompt-finetuned on a mixture of NLP tasks. METRO-T0 outperforms all similar-sized baselines on prompted NLP benchmarks, such as T0 Eval and MMLU, and rivals the state-of-the-art T0-11B model with only 8% of its parameters. Our analysis on model's neural activation and parameter sensitivity reveals that the effectiveness of METRO-T0 stems from more balanced contribution of parameters and better utilization of their capacity. The code and model checkpoints are available at https://github.com/gonglinyuan/metro_t0.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 21 May 2023.
Builds on
26 of 33 references.
References · 26
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
14 more
- 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
- 2016Pruning Filters for Efficient ConvNetscited by 1,143
- 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
- 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
- 2018A Simple Method for Commonsense Reasoningcited by 223
- 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
- 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
- 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
- 2022Your Transformer May Not be as Powerful as You Expectcited by 55
- 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
- 2020MC-BERT: Efficient Language Pre-Training via a Meta Controllercited by 15
- 2022No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Modelscited by 15
- 2022Pretraining Text Encoders with Adversarial Mixture of Training Signal Generatorscited by 9
- 2023Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memoriescited by 6
Built on by
Citing papers per year: 2024: 1.
1 paper.