Data to 14 Oct 2026

Which Attention Heads Matter for In-Context Learning?

arXiv 2502.14010 ↗First posted 19 Feb 2025cs.LG cs.AI cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exhibit impressive in-context learning (ICL) capability, enabling them to perform new tasks using only a few demonstrations in the prompt. Two different mechanisms have been proposed to explain ICL: induction heads that find and copy relevant tokens, and function vector (FV) heads whose activations compute a latent encoding of the ICL task. To better understand which of the two distinct mechanisms drives ICL, we study and compare induction heads and FV heads in 12 language models. Through detailed ablations, we discover that few-shot ICL performance depends primarily on FV heads, especially in larger models. In addition, we uncover that FV and induction heads are connected: many FV heads start as induction heads during training before transitioning to the FV mechanism. This leads us to speculate that induction facilitates learning the more complex FV mechanism that ultimately drives ICL.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 34; 2026: 27.

61 papers.

Sort:

By area:

2025Emergent Abilities in Large Language Models: A Surveycited by 37
2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
2025The Dual-Route Model of Inductioncited by 24
2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
2025Do different prompting methods yield a common task representation in language models?cited by 11
2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 10
2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
2025Mechanistic evaluation of Transformers and state space modelscited by 9
2025Just-in-time and distributed task representations in language modelscited by 9

38 more

2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
2025Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attentioncited by 8
2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
2026Causality neq Invariance: Function and Concept Vectors in LLMscited by 8
2025Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correctioncited by 7
2025Predicting the Emergence of Induction Heads in Language Model Pretrainingcited by 7
2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
2025Distinct Computations Emerge From Compositional Curricula in In-Context Learningcited by 5
2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 5
2025In-Context Algebracited by 5
2026ReAttn: Improving Attention-based Re-ranking via Attention Re-weightingcited by 5
2026Large Vision-Language Models Get Lost in Attentioncited by 5
2025Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descentcited by 4
2026Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Functioncited by 4
2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learningcited by 3
2025Latent Concept Disentanglement in Transformer-based Language Modelscited by 3
2025Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insightcited by 3
2026Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledgecited by 3
2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 3
2026Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agentscited by 3
2025Function Induction and Task Generalization: An Interpretability Study with Off-by-One Additioncited by 2
2025DFAMS: Dynamic-flow guided Federated Alignment based Multi-prototype Searchcited by 2
2025Is In-Context Learning Learning?cited by 2
2025Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learningcited by 2
2025Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysiscited by 2
2026Compressed Sensing for Capability Localization in Large Language Modelscited by 2
2026HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Headscited by 2
2026Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalizationcited by 2
2026Function-Vector Heads Are Two Populations: Writers and Cancellers in In-Context Learningcited by 2
2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inferencecited by 2
2025A circuit for predicting hierarchical structure in-context in Large Language Modelscited by 1
2026Explaining Attention with Program Synthesiscited by 1
2025Decomposing Prediction Mechanisms for In-Context Recallcited by 0
2025In-Context Learning Without Copyingcited by 0
2025The brain-AI convergence: Predictive and generative world models for general-purpose computationcited by 0
2025MIDUS: Memory-Infused Depth Up-Scalingcited by 0
2026Emergent Structured Representations Support Flexible In-Context Inference in Large Language Modelscited by 0
2026Temporal Dependencies in In-Context Learning: The Role of Induction Headscited by 0

50 most cited of 61 All 61 (JSON)