mlsanityData to 7 Oct 2026

The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis

arXiv 2311.00237 ↗First posted 1 Nov 2023cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Understanding in-context learning (ICL) capability that enables large language models (LLMs) to excel in proficiency through demonstration examples is of utmost importance. This importance stems not only from the better utilization of this capability across various tasks, but also from the proactive identification and mitigation of potential risks, including concerns regarding truthfulness, bias, and toxicity, that may arise alongside the capability. In this paper, we present a thorough survey on the interpretation and analysis of in-context learning. First, we provide a concise introduction to the background and definition of in-context learning. Then, we give an overview of advancements from two perspectives: 1) a theoretical perspective, emphasizing studies on mechanistic interpretability and delving into the mathematical foundations behind ICL; and 2) an empirical perspective, concerning studies that empirically analyze factors associated with ICL. We conclude by highlighting the challenges encountered and suggesting potential avenues for future research. We believe that our work establishes the basis for further exploration into the interpretation of in-context learning. Additionally, we have created a repository containing the resources referenced in our survey.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 62 of its 89 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

62 of 89 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
38 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2022In-context Learning and Induction Headscited by 930
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  10. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  11. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  12. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  13. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  14. 2023Larger language models do in-context learning differentlycited by 386
  15. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  16. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  17. 2023In-Context Learning Creates Task Vectorscited by 302
  18. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  19. 2022UL2: Unifying Language Learning Paradigmscited by 292
  20. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  21. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  22. 2022Active Example Selection for In-Context Learningcited by 195
  23. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  24. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  25. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  26. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  27. 2023Can Large Language Models Infer Causation from Correlation?cited by 142
  28. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  29. 2023The Learnability of In-Context Learningcited by 126
  30. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  31. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  32. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  33. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  34. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  35. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  36. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  37. 2023In-Context Learning through the Bayesian Prismcited by 78
  38. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 11.

34 papers.

Sort:

By area:

  1. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  2. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 44
  3. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  4. 2024Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspectivecited by 10
  5. 2024Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoringcited by 8
  6. 2025Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Timecited by 8
  7. 2026From Weights to Activations: Is Steering the Next Frontier of Adaptation?cited by 8
  8. 2024An Automated Explainable Educational Assessment System Built on LLMscited by 5
  9. 2025Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Modelscited by 5
  10. 2024Cascading Large Language Models for Salient Event Graph Generationcited by 3
  11. 2025Modeling Subjectivity in Cognitive Appraisal with Language Modelscited by 3
  12. 2025Detecting Actionable Requests and Offers on Social Media During Crises Using LLMscited by 3
22 more
  1. 2024Identifying and Analyzing Performance-Critical Tokens in Large Language Modelscited by 2
  2. 2025CoT-ICL Lab: A Synthetic Framework for Studying Chain-of-Thought Learning from In-Context Demonstrationscited by 2
  3. 2025Is In-Context Learning Learning?cited by 2
  4. 2026In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflictscited by 2
  5. 2026Towards Self-Improving Error Diagnosis in Multi-Agent Systemscited by 2
  6. 2025Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commercecited by 1
  7. 2025Brewing Knowledge in Context: Distillation Perspectives on In-Context Learningcited by 1
  8. 2025Surprise Calibration for Better In-Context Learningcited by 1
  9. 2025Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusionscited by 1
  10. 2025Multi-Layer Attention is the Amplifier of Demonstration Effectivenesscited by 1
  11. 2025AIC-VDS: Attention-Based In-Context Learning for Joint Velocity Control and Data Collection Scheduling in Multi-UAV-Assisted Pipeline Monitoringcited by 1
  12. 2025A Simple Generalisation of the Implicit Dynamics of In-Context Learningcited by 1
  13. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 1
  14. 2025Strategic Prompt Pricing for AIGC Services: A User-Centric Approachcited by 0
  15. 2025PictSure: Pretraining Embeddings Matters for In-Context Learning Image Classifierscited by 0
  16. 2026When Does Context Help? Error Dynamics of Contextual Information in Large Language Modelscited by 0
  17. 2026Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learningcited by 0
  18. 2026Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classificationcited by 0
  19. 2026Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learningcited by 0
  20. 2026Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifierscited by 0
  21. 2026CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMscited by 0
  22. 2026The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Eracited by 0