mlsanityData to 14 Oct 2026

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

arXiv 2305.16938 ↗First posted 26 May 2023cs.CLcited by 136

Not on the map. Why?

Abstract (arXiv)

Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning due to its simplicity and improved out-of-domain generalization, and because extensive evidence shows that fine-tuned models pick up on spurious correlations. Unfortunately, previous comparisons of the two approaches were done using models of different sizes. This raises the question of whether the observed weaker out-of-domain generalization of fine-tuned models is an inherent property of fine-tuning or a limitation of the experimental setup. In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B. Our results show that fine-tuned language models can in fact generalize well out-of-domain. We find that both approaches generalize similarly; they exhibit large variation and depend on properties such as model size and the number of examples, highlighting that robust task adaptation remains a challenge.

Key facts. Cited by 136 papers in the map (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

42 of 54 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
30 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  12. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  13. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  14. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  15. 2022Prompting GPT-3 To Be Reliablecited by 260
  16. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  17. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  18. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  19. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  20. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  21. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  22. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  23. 2022Linear Connectivity Reveals Generalization Strategiescited by 56
  24. 2019Natural Language Understanding with the Quora Question Pairs Datasetcited by 51
  25. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  26. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  27. 2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
  28. 2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
  29. 2022Lexical Generalization Improves with Larger Models and Longer Trainingcited by 9
  30. 2021∞-former: Infinite Memory Transformercited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 52; 2025: 43; 2026: 25.

136 papers.

Sort:

By area:

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2024Many-Shot In-Context Learningcited by 206
  3. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  4. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  5. 2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
  6. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 55
  7. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 53
  8. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 50
  9. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 50
  10. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 46
  11. 2024Language Models for Text Classification: Is In-Context Learning Enough?cited by 40
  12. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
38 more
  1. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  2. 2023User Inference Attacks on Large Language Modelscited by 36
  3. 2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 36
  4. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  5. 2024LLMParser: An Exploratory Study on Using Large Language Models for Log Parsingcited by 32
  6. 2023Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learningcited by 31
  7. 2024Reliable LLM-based User Simulator for Task-Oriented Dialogue Systemscited by 25
  8. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  9. 2024Leveraging Vision Language Models for Specialized Agricultural Taskscited by 25
  10. 2024MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Predictioncited by 24
  11. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 21
  12. 2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 20
  13. 2024Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical studycited by 19
  14. 2023Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Examscited by 18
  15. 2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performancecited by 18
  16. 2024Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentationcited by 17
  17. 2024Do Large Language Models Need a Content Delivery Network?cited by 16
  18. 2024Q-Probe: A Lightweight Approach to Reward Maximization for Language Modelscited by 15
  19. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 15
  20. 2024ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understandingcited by 15
  21. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  22. 2025POPri: Private Federated Learning using Preference-Optimized Synthetic Datacited by 14
  23. 2023Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMscited by 13
  24. 2024The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysiscited by 13
  25. 2024PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningcited by 13
  26. 2025SAKE: Steering Activations for Knowledge Editingcited by 13
  27. 2023Large Language Models are Miscalibrated In-Context Learnerscited by 12
  28. 2024LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Surveycited by 12
  29. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  30. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  31. 2023Measuring the Robustness of NLP Models to Domain Shiftscited by 11
  32. 2023Ada-Instruct: Adapting Instruction Generators for Complex Reasoningcited by 11
  33. 2024Large Language Models for Social Networks: Applications, Challenges, and Solutionscited by 11
  34. 2024LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domaincited by 10
  35. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
  36. 2024DRUM: Learning Demonstration Retriever for Large MUlti-modal Modelscited by 10
  37. 2025When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMscited by 10
  38. 2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9

50 most cited of 136 All 136 (JSON)