mlsanityData to 7 Oct 2026

Muppet: Massive Multi-task Representations with Pre-Finetuning

arXiv 2101.11038 ↗First posted 26 Jan 2021cs.CL cs.LGcited by 192

Not on the map. Why?

Abstract (arXiv)

We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g.~RoBERTa) and generation models (e.g.~BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.

Key facts. Cited by 192 papers in the map (data through 7 Oct 2026). Builds on 48 of its 75 references that are in the map. First posted on arXiv 26 Jan 2021.

Builds on

48 of 75 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
36 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2015Teaching Machines to Read and Comprehendcited by 1,039
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  12. 2018Neural Network Acceptability Judgmentscited by 952
  13. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  14. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  15. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  16. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  17. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  18. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  19. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  20. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  21. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  22. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  23. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  24. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  25. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  26. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  27. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  28. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  29. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  30. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  31. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  32. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  33. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  34. 2020Adversarial Training for Large Neural Language Modelscited by 109
  35. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  36. 2019This Email Could Save Your Life: Introducing the Task of Email Subject Line Generationcited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 71; 2023: 56; 2024: 26; 2025: 11; 2026: 6.

192 papers.

Sort:

By area:

  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  7. 2021MetaICL: Learning to Learn In Contextcited by 450
  8. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  9. 2021Meta-learning via Language Model In-context Tuningcited by 163
  10. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  11. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  12. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
38 more
  1. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  2. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  3. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  4. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  5. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  6. 2022Fusing finetuned models for better pretrainingcited by 110
  7. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  8. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  9. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  10. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  11. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  12. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  13. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  14. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  15. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  16. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  17. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  18. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 62
  19. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  20. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  21. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  22. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  23. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  24. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  25. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  26. 2024Bridging Remote Sensors with Multisensor Geospatial Foundation Modelscited by 46
  27. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  28. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  29. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  30. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  31. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  32. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  33. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
  34. 2024Direct Judgement Preference Optimizationcited by 33
  35. 2022Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarizationcited by 29
  36. 2022Using natural language prompts for machine translationcited by 27
  37. 2021Should We Be Pre-training? An Argument for End-task Aware Training as an Alternativecited by 26
  38. 2022IndicXNLI: Evaluating Multilingual Inference for Indian Languagescited by 26

50 most cited of 192 All 192 (JSON)