mlsanityData to 14 Oct 2026

Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning

arXiv 2305.14160 ↗First posted 23 May 2023cs.CL cs.LGcited by 236

Not on the map. Why?

Cited by 33 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In-context learning (ICL) emerges as a promising capability of large language models (LLMs) by providing them with demonstration examples to perform diverse tasks. However, the underlying mechanism of how LLMs learn from the provided context remains under-explored. In this paper, we investigate the working mechanism of ICL through an information flow lens. Our findings reveal that label words in the demonstration examples function as anchors: (1) semantic information aggregates into label word representations during the shallow computation layers' processing; (2) the consolidated information in label words serves as a reference for LLMs' final predictions. Based on these insights, we introduce an anchor re-weighting method to improve ICL performance, a demonstration compression technique to expedite inference, and an analysis framework for diagnosing ICL errors in GPT2-XL. The promising applications of our findings again validate the uncovered ICL working mechanism and pave the way for future studies.

Key facts. Cited by 236 papers in the map (data through 14 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

19 of 34 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  12. 2019Are Sixteen Heads Really Better than One?cited by 816
7 more
  1. 2022Transformers learn in-context by gradient descentcited by 732
  2. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  3. 2023Larger language models do in-context learning differentlycited by 394
  4. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  5. 2023Compositional Exemplars for In-context Learningcited by 144
  6. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  7. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 107; 2025: 76; 2026: 37.

236 papers.

Sort:

By area:

  1. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  4. 2023Function Vectors in Large Language Modelscited by 308
  5. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  6. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  7. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  8. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
  9. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 98
  10. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  11. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  12. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
38 more
  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  2. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 74
  3. 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakagecited by 69
  4. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  5. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 58
  6. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  7. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
  8. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  9. 2024Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden Statescited by 52
  10. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 52
  11. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspectivecited by 48
  12. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 48
  13. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  14. 2024Navigating the OverKill in Large Language Modelscited by 44
  15. 2024Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semanticscited by 44
  16. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  17. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  18. 2025Context-Alignment: Activating and Enhancing LLM Capabilities in Time Seriescited by 39
  19. 2024From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Taskscited by 38
  20. 2024Temporal Reasoning Transfer from Text to Videocited by 38
  21. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  22. 2025Emergent Abilities in Large Language Models: A Surveycited by 37
  23. 2024The broader spectrum of in-context learningcited by 35
  24. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 35
  25. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 35
  26. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  27. 2024Backward Lens: Projecting Language Model Gradients into the Vocabulary Spacecited by 32
  28. 2024UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulationcited by 31
  29. 2024Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysiscited by 31
  30. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 29
  31. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 29
  32. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  33. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layercited by 27
  34. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  35. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  36. 2024Instance-adaptive Zero-shot Chain-of-Thought Promptingcited by 26
  37. 2025The Dual-Route Model of Inductioncited by 24
  38. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23

50 most cited of 236 All 236 (JSON)