mlsanityData to 14 Oct 2026

Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism

arXiv 2407.17011 ↗First posted 24 Jul 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exhibit remarkable in-context learning (ICL) capabilities. However, the underlying working mechanism of ICL remains poorly understood. Recent research presents two conflicting views on ICL: One emphasizes the impact of similar examples in the demonstrations, stressing the need for label correctness and more shots. The other attributes it to LLMs' inherent ability of task recognition, deeming label correctness and shot numbers of demonstrations as not crucial. In this work, we provide a Two-Dimensional Coordinate System that unifies both views into a systematic framework. The framework explains the behavior of ICL through two orthogonal variables: whether similar examples are presented in the demonstrations (perception) and whether LLMs can recognize the task (cognition). We propose the peak inverse rank metric to detect the task recognition ability of LLMs and study LLMs' reactions to different definitions of similarity. Based on these, we conduct extensive experiments to elucidate how ICL functions across each quadrant on multiple representative classification tasks. Finally, we extend our analyses to generation tasks, showing that our coordinate system can also be used to interpret ICL for generation tasks effectively.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

34 of 47 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
22 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  2. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  3. 2023Larger language models do in-context learning differentlycited by 394
  4. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  5. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  6. 2018Hate Speech Dataset from a White Supremacy Forumcited by 198
  7. 2022In-context Examples Selection for Machine Translationcited by 172
  8. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  9. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  10. 2023Compositional Exemplars for In-context Learningcited by 144
  11. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  12. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  13. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  14. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  15. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  16. 2023In-Context Learning through the Bayesian Prismcited by 83
  17. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  18. 2022WeLM: A Well-Read Pre-trained Language Model for Chinesecited by 19
  19. 2024How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learningcited by 18
  20. 2024Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?cited by 17
  21. 2024The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysiscited by 13
  22. 2024Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learningcited by 9

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  2. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  3. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMscited by 17
  4. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  5. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 7
  6. 2025Explicit Knowledge-Guided In-Context Learning for Early Detection of Alzheimer's Diseasecited by 0