Data to 7 Oct 2026

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

arXiv 2106.13353 ↗First posted 24 Jun 2021cs.CL cs.LGcited by 139

Not on the map. Why?

Abstract (arXiv)

Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced: finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). Builds on 23 of its 26 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

23 of 26 references.

References · 23

2020Language Models are Few-Shot Learnerscited by 32,199
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234

11 more

2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
2021GPT Understands, Toocited by 715
2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
2021True Few-Shot Learning with Language Modelscited by 343
2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
2021How Many Data Points is a Prompt Worth?cited by 197
2019Argumentative Relation Classification as Plausibility Rankingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 41; 2023: 34; 2024: 31; 2025: 10; 2026: 6.

139 papers.

Sort:

By area:

2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
2021Reframing Instructional Prompts to GPTk's Languagecited by 176

38 more

2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
2022A Review on Language Models as Knowledge Basescited by 156
2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuningcited by 82
2023Unleashing the potential of prompt engineering for large language modelscited by 72
2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 56
2022FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarizationcited by 49
2024LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editingcited by 47
2022DeepStruct: Pretraining of Language Models for Structure Predictioncited by 45
2023The language of prompting: What linguistic properties make a prompt successful?cited by 43
2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
2023Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysiscited by 35
2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
2021Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Trainingcited by 28
2023Prompt Learning for News Recommendationcited by 25
2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
2022Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibrationcited by 24
2023Understanding Emergent In-Context Learning from a Kernel Regression Perspectivecited by 23
2022Pre-trained Language Models Can be Fully Zero-Shot Learnerscited by 22
2021Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biasescited by 21
2021Exploring Prompt-based Few-shot Learning for Grounded Dialog Generationcited by 20
2021A Few More Examples May Be Worth Billions of Parameterscited by 19
2023Tree Prompting: Efficient Task Adaptation without Fine-Tuningcited by 17
2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performancecited by 17
2024Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Studycited by 17
2023Prompt Engineering Through the Lens of Optimal Controlcited by 16

50 most cited of 139 All 139 (JSON)