mlsanityData to 7 Oct 2026

Making Pre-trained Language Models Better Few-shot Learners

arXiv 2012.15723 ↗First posted 31 Dec 2020cs.CL cs.LGcited by 1,255

Not on the map. Why?

Cited by 62 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF--better few-shot fine-tuning of language models--a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.

Key facts. Cited by 1,255 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 126; 2022: 305; 2023: 406; 2024: 245; 2025: 131; 2026: 42.

1,255 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
38 more
  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2022In-context Learning and Induction Headscited by 930
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2023Large Language Models as Optimizerscited by 830
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  13. 2021GPT Understands, Toocited by 715
  14. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  15. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  16. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  17. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  18. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  19. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  20. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  21. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  22. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  23. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  24. 2023Cognitive Architectures for Language Agentscited by 422
  25. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  26. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  27. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  28. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  29. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  30. 2021True Few-Shot Learning with Language Modelscited by 343
  31. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  32. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  33. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  34. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  35. 2022Large Language Models Are Reasoning Teacherscited by 318
  36. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  37. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  38. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291

50 most cited of 1,255 All 1,255 (JSON)