# True Few-Shot Learning with Prompts -- A Real-World Perspective

arXiv: https://arxiv.org/abs/2111.13440 · HTML: https://mlsanity.com/paper/2111.13440/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 94 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 30 of 66)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)

## Built on by (most cited first, 30 of 38)

- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Opportunities and Challenges in Neural Dialog Tutoring](https://mlsanity.com/paper/2301.09919/) (2023)
- [ToMChallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mind](https://mlsanity.com/paper/2305.15068/) (2023)
- [Co-training Improves Prompt-based Learning for Large Language Models](https://mlsanity.com/paper/2202.00828/) (2022)
- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [Instruction Tuned Models are Quick Learners](https://mlsanity.com/paper/2306.05539/) (2023)
- [On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations](https://mlsanity.com/paper/2204.12708/) (2022)
- [Evaluating the Robustness of Discrete Prompts](https://mlsanity.com/paper/2302.05619/) (2023)
- [Federated Few-Shot Learning for Mobile NLP](https://mlsanity.com/paper/2212.05974/) (2022)
- [BioTABQA: Instruction Learning for Biomedical Table Question Answering](https://mlsanity.com/paper/2207.02419/) (2022)
- [Automated Few-shot Classification with Instruction-Finetuned Language Models](https://mlsanity.com/paper/2305.12576/) (2023)
- [Large Language Models Vote: Prompting for Rare Disease Identification](https://mlsanity.com/paper/2308.12890/) (2023)
- [Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning](https://mlsanity.com/paper/2310.08309/) (2023)
- [Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors](https://mlsanity.com/paper/2510.09536/) (2025)
- [Task Bias in Vision-Language Models](https://mlsanity.com/paper/2212.04412/) (2022)
- [Weaker Than You Think: A Critical Look at Weakly Supervised Learning](https://mlsanity.com/paper/2305.17442/) (2023)
- [Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding](https://mlsanity.com/paper/2306.11066/) (2023)
- [Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMs](https://mlsanity.com/paper/2411.18564/) (2024)
- [Towards Practical Few-shot Federated NLP](https://mlsanity.com/paper/2212.00192/) (2022)
- [The Benefits of Label-Description Training for Zero-Shot Text Classification](https://mlsanity.com/paper/2305.02239/) (2023)
- [Exploring Prompt Engineering: A Systematic Review with SWOT Analysis](https://mlsanity.com/paper/2410.12843/) (2024)
- [Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages](https://mlsanity.com/paper/2507.03003/) (2025)
- [Bias in, Bias out: Annotation Bias in Multilingual Large Language Models](https://mlsanity.com/paper/2511.14662/) (2025)
- [Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models](https://mlsanity.com/paper/2310.13998/) (2023)
- [Deconstructing In-Context Learning: Understanding Prompts via Corruption](https://mlsanity.com/paper/2404.02054/) (2024)
- [A Multi-solution Study on GDPR AI-enabled Completeness Checking of DPAs](https://mlsanity.com/paper/2311.13881/) (2023)
- [Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting](https://mlsanity.com/paper/2403.13369/) (2024)
- [Do LLMs Think Fast and Slow? A Causal Study on Sentiment Analysis](https://mlsanity.com/paper/2404.11055/) (2024)

## Abstract (arXiv)

Prompt-based approaches are strong at few-shot learning. However, Perez et al. (2021) have recently cast doubt on their performance because they had difficulty getting good results in a "true" few-shot setting in which prompts and hyperparameters cannot be tuned on a dev set. In view of this, we conduct an extensive study of PET, a method that combines textual instructions with example-based finetuning. We show that, if correctly configured, PET performs strongly in a true few-shot setting, i.e., without a dev set. Crucial for this strong performance is PET's ability to intelligently handle multiple prompts. We then put our findings to a real-world test by running PET on RAFT, a benchmark of tasks taken directly from realistic NLP applications for which no labeled dev or test sets are available. PET achieves a new state of the art on RAFT and performs close to non-expert humans for 7 out of 11 tasks. These results demonstrate that prompt-based learners like PET excel at true few-shot learning and underpin our belief that learning from instructions will play an important role on the path towards human-like few-shot learning capabilities.