mlsanityData to 7 Oct 2026

ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization

arXiv 2201.06910 ↗First posted 18 Jan 2022cs.LG cs.CLcited by 54

Not on the map. Why?

Abstract (arXiv)

We propose a multitask pretraining approach ZeroPrompt for zero-shot generalization, focusing on task scaling and zero-shot prompting. While previous models are trained on only a few dozen tasks, we scale to 1,000 tasks for the first time using real-world data. This leads to a crucial discovery that task scaling can be an efficient alternative to model scaling; i.e., the model size has little impact on performance with an extremely large number of tasks. Our results show that task scaling can substantially improve training efficiency by 30 times in FLOPs. Moreover, we present a prompting method that incorporates a genetic algorithm to automatically search for the best prompt for unseen tasks, along with a few other improvements. Empirically, ZeroPrompt substantially improves both the efficiency and the performance of zero-shot learning across a variety of academic and production datasets.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 18 Jan 2022.

Builds on

33 of 43 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2019Invariant Risk Minimizationcited by 1,832
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
21 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  2. 2020In Search of Lost Domain Generalizationcited by 1,042
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  6. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  7. 2021How Many Data Points is a Prompt Worth?cited by 197
  8. 2021Generating Datasets with Pretrained Language Modelscited by 192
  9. 2021SelfReg: Self-supervised Contrastive Regularization for Domain Generalizationcited by 189
  10. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  11. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  12. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  13. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  14. 2019Domain Agnostic Learning with Disentangled Representationscited by 107
  15. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  16. 2020Cross-domain Face Presentation Attack Detection via Multi-domain Disentangled Representation Learningcited by 93
  17. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  18. 2021Domain Generalization via Gradient Surgerycited by 60
  19. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  20. 2020Domain2Vec: Domain Embedding for Unsupervised Domain Adaptationcited by 24
  21. 2021Conterfactual Generative Zero-Shot Semantic Segmentationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 23; 2023: 20; 2024: 8; 2025: 2; 2026: 1.

54 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  6. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  7. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  8. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  9. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  10. 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 100
  11. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  12. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
38 more
  1. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
  2. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  3. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  4. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  5. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 53
  6. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  7. 2023Augmented Large Language Models with Parametric Knowledge Guidingcited by 37
  8. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  9. 2023Chinese Open Instruction Generalist: A Preliminary Releasecited by 35
  10. 2024GenQA: Generating Millions of Instructions from a Handful of Promptscited by 35
  11. 2022Robustness of Learning from Task Instructionscited by 34
  12. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  13. 2023Interactive Natural Language Processingcited by 31
  14. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 22
  15. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  16. 2023Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluationcited by 19
  17. 2023A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Modelcited by 19
  18. 2023LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Modelscited by 17
  19. 2024Contrastive Instruction Tuningcited by 17
  20. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  21. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  22. 2023FreeAL: Towards Human-Free Active Learning in the Era of Large Language Modelscited by 14
  23. 2022Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspectivecited by 13
  24. 2024Self-Prompt Tuning: Enable Autonomous Role-Playing in LLMscited by 12
  25. 2024CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructionscited by 12
  26. 2022Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learningcited by 10
  27. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6
  28. 2022Improving Task Generalization via Unified Schema Promptcited by 6
  29. 2023Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Modelscited by 6
  30. 2023Large Language Models Need Holistically Thought in Medical Conversational QAcited by 5
  31. 2023STORYWARS: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generationcited by 5
  32. 2022Zero-Label Prompt Selectioncited by 4
  33. 2023SymNoise: Advancing Language Model Fine-tuning with Symmetric Noisecited by 4
  34. 2022FewFedWeight: Few-shot Federated Learning Framework across Multiple NLP Taskscited by 3
  35. 2025Topic Coverage-based Demonstration Retrieval for In-Context Learningcited by 3
  36. 2022Leveraging Natural Supervision for Language Representation Learning and Generationcited by 2
  37. 2022Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Promptcited by 2
  38. 2024AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuningcited by 2

50 most cited of 54 All 54 (JSON)