mlsanityData to 7 Oct 2026

Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning

arXiv 2410.04691 ↗First posted 7 Oct 2024cs.LG cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Fine-tuning and in-context learning (ICL) are two prevalent methods in imbuing large language models with task-specific knowledge. It is commonly believed that fine-tuning can surpass ICL given sufficient training samples as it allows the model to adjust its internal parameters based on the data. However, this paper presents a counterintuitive finding: For tasks with implicit patterns, ICL captures these patterns significantly better than fine-tuning. We developed several datasets featuring implicit patterns, such as sequences determining answers through parity or identifying reducible terms in calculations. We then evaluated the models' understanding of these patterns under both fine-tuning and ICL across models ranging from 0.5B to 7B parameters. The results indicate that models employing ICL can quickly grasp deep patterns and significantly improve accuracy. In contrast, fine-tuning, despite utilizing thousands of times more training samples than ICL, achieved only limited improvements. We also proposed circuit shift theory from a mechanistic interpretability's view to explain why ICL wins.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 34 of its 38 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

34 of 38 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2022A Survey on In-context Learningcited by 717
  8. 2024Yi: Open Foundation Models by 01.AIcited by 706
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
22 more
  1. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  2. 2018Measuring abstract reasoning in neural networkscited by 165
  3. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  4. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  5. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  6. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  7. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70
  8. 2023Large Language Models can Learn Rulescited by 66
  9. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasonerscited by 63
  10. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  11. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 52
  12. 2023Large Language Models for Code Analysis: Do LLMs Really Do Their Job?cited by 44
  13. 2023Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMscited by 38
  14. 2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36
  15. 2024LLMs for Relational Reasoning: How Far are We?cited by 21
  16. 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20
  17. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  18. 2021Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Languagecited by 15
  19. 2024ItD: Large Language Models Can Teach Themselves Induction through Deductioncited by 8
  20. 2024DiLA: Enhancing LLM Tool Learning with Differential Logic Layercited by 6
  21. 2023A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Taskscited by 5
  22. 2022A Few-shot Approach to Resume Information Extraction via Promptscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 10.

20 papers.

Sort:

By area:

  1. 2025On Finetuning Tabular Foundation Modelscited by 13
  2. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  3. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 7
  4. 2025Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Modelscited by 3
  5. 2025Can Past Experience Accelerate LLM Reasoning?cited by 3
  6. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  7. 2025RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selectioncited by 2
  8. 2025Evaluating Steering Techniques using Human Similarity Judgmentscited by 2
  9. 2025Inference-Time Distillation: Cost-Efficient Agents Without Fine-Tuning or Manual Prompt Engineeringcited by 2
  10. 2025Supervised In-Context Fine-Tuning for Generative Sequence Labelingcited by 1
  11. 2025SVBRD-LLM: Self-Verifying Behavioral Rule Discovery for Autonomous Vehicle Identificationcited by 1
  12. 2026RELISH: LLM REgression with a Latent Iterative State Headcited by 1
8 more
  1. 2026Stochasticity in Tokenisation Improves Robustnesscited by 1
  2. 2026Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspectivecited by 1
  3. 2026AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reusecited by 1
  4. 2025Large Language Models Can Perform Automatic Modulation Classification via Discretized Self-supervised Candidate Retrievalcited by 0
  5. 2026Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxiescited by 0
  6. 2026The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithmscited by 0
  7. 2026Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agentscited by 0
  8. 2026When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attentioncited by 0