mlsanityData to 7 Oct 2026

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

arXiv 2111.10952 ↗First posted 22 Nov 2021cs.CL cs.LGcited by 161

Not on the map. Why?

Abstract (arXiv)

Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Builds on 104 of its 160 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

104 of 160 references.

References · 104

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  13. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  14. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  15. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  16. 2021Scaling Vision Transformerscited by 993
  17. 2018Neural Network Acceptability Judgmentscited by 952
  18. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  19. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  20. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  21. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  22. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  23. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  24. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  25. 2019ELI5: Long Form Question Answeringcited by 567
  26. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  27. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  28. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  29. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  30. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  31. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  32. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  33. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  34. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  35. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  36. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  37. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  38. 2018QuAC : Question Answering in Contextcited by 379

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 61; 2023: 57; 2024: 25; 2025: 7; 2026: 8.

161 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2022Galactica: A Large Language Model for Sciencecited by 736
  10. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
38 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  2. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  3. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  4. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  5. 2022Confident Adaptive Language Modelingcited by 297
  6. 2022UL2: Unifying Language Learning Paradigmscited by 292
  7. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  8. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  9. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  10. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  11. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  12. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  13. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  14. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  15. 2022Fusing finetuned models for better pretrainingcited by 110
  16. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  17. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  18. 2024KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graphcited by 90
  19. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  20. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  21. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  22. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  23. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  24. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  25. 2023CoEdIT: Text Editing by Task-Specific Instruction Tuningcited by 67
  26. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  27. 2023UniHCP: A Unified Model for Human-Centric Perceptionscited by 59
  28. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  29. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  30. 2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54
  31. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  32. 2023Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluationcited by 52
  33. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 45
  34. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  35. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  36. 2022LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restrictioncited by 43
  37. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  38. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42

50 most cited of 161 All 161 (JSON)