mlsanityData to 7 Oct 2026

True Few-Shot Learning with Language Models

arXiv 2105.11447 ↗First posted 24 May 2021cs.CL cs.LG stat.MLcited by 343

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.

Key facts. Cited by 343 papers in the map (data through 7 Oct 2026). Builds on 54 of its 103 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

54 of 103 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2016Matching Networks for One Shot Learningcited by 2,246
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
38 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  9. 2020In Search of Lost Domain Generalizationcited by 1,042
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  12. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  13. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  14. 2019Meta-Learning with Implicit Gradientscited by 523
  15. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  16. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  17. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  18. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  19. 2020A Metric Learning Reality Checkcited by 317
  20. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  21. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  22. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  23. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  24. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  25. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  26. 2021How Many Data Points is a Prompt Worth?cited by 197
  27. 2016Learning to Optimizecited by 195
  28. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  29. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  30. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  31. 2017Unsupervised Neural Machine Translationcited by 148
  32. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  33. 2021Entailment as Few-Shot Learnercited by 122
  34. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  35. 2015Practical Bayesian model evaluation using leave-one-out cross-validation and WAICcited by 110
  36. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  37. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  38. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 82; 2023: 102; 2024: 76; 2025: 33; 2026: 22.

343 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  8. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
38 more
  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  2. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  4. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  5. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  6. 2021MetaICL: Learning to Learn In Contextcited by 450
  7. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  8. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  9. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  10. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  11. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  12. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  13. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  14. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  15. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  16. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  17. 2022Can language models learn from explanations in context?cited by 248
  18. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  19. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  20. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  21. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  22. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  23. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  24. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  25. 2022Active Example Selection for In-Context Learningcited by 195
  26. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  27. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  28. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  29. 2021Meta-learning via Language Model In-context Tuningcited by 163
  30. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 158
  31. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  32. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  33. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  34. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  35. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  36. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  37. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  38. 2023Training Language Models with Language Feedback at Scalecited by 125

50 most cited of 343 All 343 (JSON)