mlsanityData to 7 Oct 2026

True Few-Shot Learning with Prompts -- A Real-World Perspective

arXiv 2111.13440 ↗First posted 26 Nov 2021cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

Prompt-based approaches are strong at few-shot learning. However, Perez et al. (2021) have recently cast doubt on their performance because they had difficulty getting good results in a "true" few-shot setting in which prompts and hyperparameters cannot be tuned on a dev set. In view of this, we conduct an extensive study of PET, a method that combines textual instructions with example-based finetuning. We show that, if correctly configured, PET performs strongly in a true few-shot setting, i.e., without a dev set. Crucial for this strong performance is PET's ability to intelligently handle multiple prompts. We then put our findings to a real-world test by running PET on RAFT, a benchmark of tasks taken directly from realistic NLP applications for which no labeled dev or test sets are available. PET achieves a new state of the art on RAFT and performs close to non-expert humans for 7 out of 11 tasks. These results demonstrate that prompt-based learners like PET excel at true few-shot learning and underpin our belief that learning from instructions will play an important role on the path towards human-like few-shot learning capabilities.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 66 of its 94 references that are in the map. First posted on arXiv 26 Nov 2021.

Builds on

66 of 94 references.

References · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  13. 2019How Can We Know What Language Models Know?cited by 953
  14. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  15. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  16. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  17. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  18. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  19. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  20. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  21. 2021True Few-Shot Learning with Language Modelscited by 343
  22. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  23. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  24. 2020Calibration of Pre-trained Transformerscited by 307
  25. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  26. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  27. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  28. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  29. 2021Generating Datasets with Pretrained Language Modelscited by 192
  30. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  31. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  32. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  33. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  34. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  35. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  36. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  37. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  38. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 11; 2023: 15; 2024: 8; 2025: 4.

38 papers.

Sort:

By area:

  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  2. 2022Efficient Few-Shot Learning Without Promptscited by 135
  3. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  4. 2023Opportunities and Challenges in Neural Dialog Tutoringcited by 33
  5. 2023ToMChallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mindcited by 28
  6. 2022Co-training Improves Prompt-based Learning for Large Language Modelscited by 24
  7. 2023Instruction Tuned Models are Quick Learnerscited by 21
  8. 2022A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processingcited by 20
  9. 2022On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlationscited by 19
  10. 2023Evaluating the Robustness of Discrete Promptscited by 18
  11. 2022Federated Few-Shot Learning for Mobile NLPcited by 15
  12. 2022BioTABQA: Instruction Learning for Biomedical Table Question Answeringcited by 13
26 more
  1. 2023Automated Few-shot Classification with Instruction-Finetuned Language Modelscited by 9
  2. 2023Large Language Models Vote: Prompting for Rare Disease Identificationcited by 9
  3. 2023Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learningcited by 9
  4. 2025Evaluating Robustness of Large Language Models Against Multilingual Typographical Errorscited by 9
  5. 2022Task Bias in Vision-Language Modelscited by 8
  6. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  7. 2023Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understandingcited by 8
  8. 2024Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMscited by 6
  9. 2022Towards Practical Few-shot Federated NLPcited by 5
  10. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  11. 2025Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languagescited by 5
  12. 2023The Benefits of Label-Description Training for Zero-Shot Text Classificationcited by 4
  13. 2023Transductive Learning for Textual Few-Shot Classification in API-based Embedding Modelscited by 4
  14. 2024Deconstructing In-Context Learning: Understanding Prompts via Corruptioncited by 4
  15. 2025Bias in, Bias out: Annotation Bias in Multilingual Large Language Modelscited by 4
  16. 2023A Multi-solution Study on GDPR AI-enabled Completeness Checking of DPAscited by 3
  17. 2024Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Promptingcited by 2
  18. 2024Do LLMs Think Fast and Slow? A Causal Study on Sentiment Analysiscited by 2
  19. 2024Interpretable Cross-Examination Technique (ICE-T): Using highly informative features to boost LLM performancecited by 2
  20. 2023Weakly Supervised Text Classification on Free Text Comments in Patient-Reported Outcome Measurescited by 1
  21. 2023Natural Language based Context Modeling and Reasoning for Ubiquitous Computing with Large Language Models: A Tutorialcited by 1
  22. 2023Deep Natural Language Feature Learning for Interpretable Predictioncited by 1
  23. 2024GPT-3 Powered Information Extraction for Building Robust Knowledge Basescited by 1
  24. 2025An Empirically-grounded tool for Automatic Prompt Linting and Repair: A Case Study on Bias, Vulnerability, and Optimization in Developer Promptscited by 1
  25. 2022Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Taskscited by 0
  26. 2024PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuningcited by 0