mlsanityData to 14 Oct 2026

A Survey to Recent Progress Towards Understanding In-Context Learning

arXiv 2402.02212 ↗First posted 3 Feb 2024cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

In-Context Learning (ICL) empowers Large Language Models (LLMs) with the ability to learn from a few examples provided in the prompt, enabling downstream generalization without the requirement for gradient updates. Despite encouragingly empirical success, the underlying mechanism of ICL remains unclear. Existing research remains ambiguous with various viewpoints, utilizing intuition-driven and ad-hoc technical solutions to interpret ICL. In this paper, we leverage a data generation perspective to reinterpret recent efforts from a systematic angle, demonstrating the potential broader usage of these popular technical solutions. For a conceptual definition, we rigorously adopt the terms of skill recognition and skill learning. Skill recognition selects one learned data generation function previously seen during pre-training while skill learning can learn new data generation functions from in-context data. Furthermore, we provide insights into the strengths and weaknesses of both abilities, emphasizing their commonalities through the perspective of data generation. This analysis suggests potential directions for future research.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 78 of its 110 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

78 of 110 references.

References · 78

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  10. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
38 more
  1. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  2. 2023Larger language models do in-context learning differentlycited by 394
  3. 2023Language Models can Solve Computer Taskscited by 372
  4. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  6. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  7. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  8. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  9. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  10. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  11. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 285
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  13. 2022Generating Sequences by Learning to Self-Correctcited by 277
  14. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  15. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  16. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  17. 2021User-friendly introduction to PAC-Bayes boundscited by 239
  18. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  19. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  20. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  21. 2023Large Language Models as General Pattern Machinescited by 197
  22. 2021Thinking Like Transformerscited by 184
  23. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  24. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  25. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  26. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  27. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  28. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  29. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  30. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  31. 2023The Learnability of In-Context Learningcited by 127
  32. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  33. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  34. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  35. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 113
  36. 2023How do Language Models Bind Entities in Context?cited by 108
  37. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  38. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 6.

14 papers.

Sort:

By area:

  1. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
  2. 2024Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insightscited by 44
  3. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  4. 2024On the Intrinsic Self-Correction Capability of LLMs: Uncertainty and Latent Conceptcited by 16
  5. 2024Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesiscited by 15
  6. 2024No Free Lunch for Defending Against Prefilling Attack by In-Context Learningcited by 2
  7. 2025Implicit In-Context Learning: Evidence from Artificial Language Experimentscited by 2
  8. 2025CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Modelscited by 2
  9. 2026ICI-VLA: In-Context Imitation with Spatiotemporally Aligned Demonstrations for Vision-Language-Action Modelscited by 1
  10. 2026From Exposure to Internalization: Dual-Stream Calibration for In-context Clinical Reasoningcited by 0
  11. 2026The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Eracited by 0
  12. 2026From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoringcited by 0
2 more
  1. 2026Verb-ICL: Rethinking In-Context Learning for Structured Predictioncited by 0
  2. 2026When Does Retrieval Help? A Study of In-Context Adaptation in Vision-Language-Action Modelscited by 0