mlsanityData to 7 Oct 2026

Multitask Prompted Training Enables Zero-Shot Task Generalization

arXiv 2110.08207 ↗First posted 15 Oct 2021cs.LG cs.CLcited by 1,462

Not on the map. Why?

Cited by 117 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.

Key facts. Cited by 1,462 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Oct 2021.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 274; 2023: 563; 2024: 366; 2025: 175; 2026: 73.

1,462 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
38 more
  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  13. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  14. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  15. 2023LIMA: Less Is More for Alignmentcited by 1,080
  16. 2022Editing Models with Task Arithmeticcited by 1,076
  17. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  18. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  19. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  20. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  21. 2023Teaching Large Language Models to Self-Debugcited by 963
  22. 2023BloombergGPT: A Large Language Model for Financecited by 923
  23. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  24. 2023A Survey on Multimodal Large Language Modelscited by 898
  25. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  26. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  27. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  28. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  29. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  30. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  31. 2022A Generalist Agentcited by 802
  32. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  33. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  34. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  35. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  36. 2022Galactica: A Large Language Model for Sciencecited by 736
  37. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  38. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733

50 most cited of 1,462 All 1,462 (JSON)