mlsanityData to 14 Oct 2026

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

arXiv 2205.05638 ↗First posted 11 May 2022cs.LG cs.AI cs.CLcited by 896

Not on the map. Why?

Cited by 34 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Few-shot in-context learning (ICL) enables pre-trained language models to perform a previously-unseen task without any gradient-based training by feeding a small number of training examples as part of the input. ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made. Parameter-efficient fine-tuning (PEFT) (e.g. adapter modules, prompt tuning, sparse update methods, etc.) offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task. In this paper, we rigorously compare few-shot ICL and PEFT and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs. Along the way, we introduce a new PEFT method called (IA)3 that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters. We also propose a simple recipe based on the T0 model called T-Few that can be applied to new tasks without task-specific tuning or modifications. We validate the effectiveness of T-Few on completely unseen tasks by applying it to the RAFT benchmark, attaining super-human performance for the first time and outperforming the state-of-the-art by 6% absolute. All of the code used in our experiments is publicly available.

Key facts. Cited by 896 papers in the map (data through 14 Oct 2026). Builds on 65 of its 98 references that are in the map. First posted on arXiv 11 May 2022.

Builds on

65 of 98 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2022Visual Prompt Tuningcited by 1,681
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2021Learning to Prompt for Continual Learningcited by 944
  7. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  11. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  13. 2017Learning multiple visual domains with residual adapterscited by 557
  14. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  15. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  16. 2021MetaICL: Learning to Learn In Contextcited by 454
  17. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  18. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  19. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  20. 2021True Few-Shot Learning with Language Modelscited by 344
  21. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  22. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  23. 2022Can language models learn from explanations in context?cited by 249
  24. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  25. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  26. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  27. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  28. 2021How Many Data Points is a Prompt Worth?cited by 198
  29. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  30. 2021Meta-learning via Language Model In-context Tuningcited by 164
  31. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  32. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  33. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  34. 2021The Efficiency Misnomercited by 98
  35. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  36. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  37. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  38. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 40; 2023: 189; 2024: 321; 2025: 193; 2026: 153.

896 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  5. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  6. 2024A Survey on Large Language Models for Code Generationcited by 658
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  8. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  9. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  10. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  11. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  12. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
38 more
  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2023A Survey on Model Compression for Large Language Modelscited by 269
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  4. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  5. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  6. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  7. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  8. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  9. 2024Many-Shot In-Context Learningcited by 206
  10. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  11. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  12. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  13. 2023Efficient Large Language Models: A Surveycited by 186
  14. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  15. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  16. 2022In-context Examples Selection for Machine Translationcited by 172
  17. 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 169
  18. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  19. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 159
  20. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  21. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  22. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  23. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  24. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  25. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  26. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  27. 2022Efficient Few-Shot Learning Without Promptscited by 137
  28. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  29. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  30. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  31. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  32. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  33. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  34. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  35. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  36. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  37. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  38. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114

50 most cited of 896 All 896 (JSON)