mlsanityData to 7 Oct 2026

How Many Data Points is a Prompt Worth?

arXiv 2103.08493 ↗First posted 15 Mar 2021cs.LGcited by 197

Not on the map. Why?

Abstract (arXiv)

When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 15 Mar 2021.

Builds on

23 of 33 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  12. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
11 more
  1. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  2. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  3. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  4. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  5. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  6. 2018A Simple Method for Commonsense Reasoningcited by 223
  7. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  8. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  9. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  10. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  11. 2019Zero-shot Text Classification With Generative Language Modelscited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 35; 2022: 76; 2023: 45; 2024: 28; 2025: 11; 2026: 2.

197 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  9. 2021True Few-Shot Learning with Language Modelscited by 343
  10. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  11. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  12. 2022Can language models learn from explanations in context?cited by 248
38 more
  1. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  2. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  3. 2022Is GPT-3 a Good Data Annotator?cited by 208
  4. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  5. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  6. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  7. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  8. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  9. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  10. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  11. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  12. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  13. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  14. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 116
  15. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  16. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  17. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  18. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  19. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  20. 2023Effective Prompt Extraction from Language Modelscited by 83
  21. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  22. 2021Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractioncited by 73
  23. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  24. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 67
  25. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  26. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  27. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  28. 2022Legal Prompting: Teaching a Language Model to Think Like a Lawyercited by 60
  29. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 60
  30. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  31. 2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
  32. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  33. 2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54
  34. 2021A Survey on Green Deep Learningcited by 52
  35. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  36. 2022NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Flycited by 48
  37. 2022Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extractioncited by 40
  38. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40

50 most cited of 197 All 197 (JSON)