mlsanityData to 7 Oct 2026

It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

arXiv 2009.07118 ↗First posted 15 Sep 2020cs.CL cs.AI cs.LGcited by 668

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.

Key facts. Cited by 668 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 121; 2022: 208; 2023: 166; 2024: 99; 2025: 48; 2026: 26.

668 papers.

Sort:

By area:

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2021GPT Understands, Toocited by 715
  12. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
38 more
  1. 2022Large Language Models Can Self-Improvecited by 656
  2. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  5. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  7. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  8. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  9. 2021True Few-Shot Learning with Language Modelscited by 343
  10. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  11. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  13. 2022Large Language Models Are Reasoning Teacherscited by 318
  14. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  15. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  16. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  17. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  18. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  19. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  20. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  21. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  22. 2022Prompt Distribution Learningcited by 235
  23. 2022Data Contamination: From Memorization to Exploitationcited by 234
  24. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  25. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  26. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  27. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  28. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  29. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  30. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  31. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  32. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  33. 2021How Many Data Points is a Prompt Worth?cited by 197
  34. 2021Generating Datasets with Pretrained Language Modelscited by 192
  35. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  36. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  37. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  38. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182

50 most cited of 668 All 668 (JSON)